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Executive Summary 


In January 2015, the Northwest Board of Cooperative Educational Services (NW BOCES) received 
a five-year Investing in Innovation (13) grant to develop and implement the System for Educator 
Effectiveness Development (SEED) program—an innovative professional development (PD) 
system designed to provide geographically isolated educators an impactful tool to improve teacher 
effectiveness. McREL International (McREL) was contracted by NW BOCES to conduct an 
independent evaluation of SEED. 


Twenty-one schools in rural areas, where obtaining PD can be a challenge, participated in SEED 
between 2015 and 2020. (One of them closed during the study period but the majority of its 
students transferred to other participating schools.) Because SEED was a school-level intervention, 
the target population consisted of all principals, assistant principals, teachers, and students in the 
participating sites. 


SEED had six goals. They follow, along with our main findings related to each: 


Goal One: Successfully implement and continually improve SEED in up to 30 NW BOCES 
schools during a four-year period. Some early technological challenges were rapidly overcome, 
and implementation proceeded largely as planned in all 21 schools. SEED’s three components 
(creation, delivery, and participation) were implemented at a generally adequate level and with 
planned rigor: Evidence-based PD courses driven by teacher needs and interests were created, 
followed by an online/blended delivery platform and an online resource repository (i.e., SEED 
Personalized Accessible Knowledge [SEED PAK]). Teacher participation in SEED courses through 
the online and blended delivery was a success with high marks to the quality of SEED courses— 
average ratings ranged from 5.32 to 5.50 on a 6-point scale. Teachers also gave SEED PAK high 
scores for quality—average ratings ranged from 4.19 to 5.32 on a 6-point scale; yet, the level of 
SEEK PAK usage was lower than expected. Project staff suspected that the low usage of SEED 
PAK might due to teachers' preference for human connection while learning. 


Goal Two: Increase principal engagement in teacher professional growth and support 
throughout the project period. Principal engagement (as gauged by teacher and principal surveys) 
grew markedly over the three years. This was true in both high participation and low participation 
schools.’ While causation could not be determined in this study design, these results suggest any 
teacher in a SEED school, regardless of their personal degree of participation, stands to benefit by 
having a principal who has grown more attuned to their needs for PD. 


Goal Three: Increase rural teachers’ access to and use of current best practices and up-to- 
date content knowledge. Principals agreed that the PD made available to their teachers as a result 
of SEED was up-to-date and evidence-based—and their level of agreement rose over time. When 
asked about the extent to which schools offered time and resources to access PD, both principals’ 
and teachers’ ratings increased significantly over time. Teachers who used SEED were likelier than 


' School level SEED participation rates were calculated based on teacher participation data collected in Y3 (2018-19). 
Schools with equal or more than 50% of teachers who participated in SEED were coded as high-participation schools, 
while schools with less than 50% of teachers who participated in SEED were coded as low-participation schools. 
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those who didn’t, to say they had access to PD that was up-to-date, evidence-based, data-driven, and 
easy to access. 


Goal Four: Support teachers in successfully implementing SEED practices and content. 
Several outcomes were used to measure this goal and all of them showed statistically significant 
increases over time. Additionally, SEED teachers reported more favorable outcomes compared to 
non-SEED teachers on measures related to knowledge of best practices, though the two groups did 
not differ in terms of change in classroom practice, such as teacher evaluation ratings and 
opportunities to collaborate with peers. Similar findings were found when comparing the measures 
between high-participation and low-participation schools. Due to the study design, causal inference 
could not be established. Yet, these findings are encouraging as they suggest that SEED 
participation at the teacher and school levels was associated with teachers’ knowledge of best 
practices and positive change in classroom practices (i.e., teacher perception of SEED impact on 
their practices). 


Goal 5: Improve student achievement. The ultimate goal of SEED was to increase student 
achievement, and a quasi-experimental study provided some promising evidence that it did. Of six 
student outcomes we assessed, SEED had a statistically significant positive effect on one (minority 
students’ English language acquisition, or ELA) and marginally significant positive effects on four 
(all related to English language acquisition and math outcomes among racial/ethnic minority and 
free and reduced-lunch, or FRL, students). Some caution is advised in interpreting these findings too 
broadly, as the Colorado Department of Education masks data for subgroups smaller than 16, 
meaning the subgroups that were available for us to analyze didn’t fully represent SEED schools. 


Goal 6: Improve student engagement. Student engagement tends to decrease as students age, and 
this well-documented phenomenon held true in SEED schools as well. We did, however, establish 
that SEED was associated with less loss of engagement by comparing high-participation schools 
with low-participation schools. While causation cannot be made due to the nature of the 
correlational design, the positive association between level of SEED participation and the student 
engagement measure was encouraging. 


Taken together, over one year of planning and pilot testing (2015-16) and four years of full 
implementation (2016-17 to 2019-20), SEED achieved all six of its goals. 
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Introduction 


In January 2015, the Northwest Board of Cooperative Educational Services (NW BOCES) received 
a five-year Investing in Innovation (i3)* grant to develop and implement the System for Educator 
Effectiveness Development (SEED) program—an innovative professional development (PD) 
system designed to provide geographically isolated educators an impactful tool to improve teacher 
effectiveness. 


NW BOCES serves seven school districts (30 schools) with 87% (six out of seven) located in rural 
areas. Rural areas often have a disproportionate number of new teachers who need additional 
support as they learn about the profession and their own practices. Additionally, rural teachers are 
often not adequately prepared to provide supports for high needs students (Gaumer Erickson, 
Noonan, & McCall, 2012). In the districts served by NW BOCES, about one-third of students 
(32%) are low-income; 11% are in special education; and 8% are English learners. The need for 
evidence-based PD to support educators within NW BOCES is evident. 


Literature Review 


Definitions of evidence-based PD. Evidence-based PD refers to structured professional learning 
that improves teacher practices and student learning (Darling-Hammond, Hyler, & Gardner, 2017). 
Accumulated literature has indicated that effective PD often incorporates elements that support peer 
collaboration and engage educators in collective inquiry through a community of learners (Darling- 
Hammond et al., 2017; Jaquith, Mindich, Wei, & Darling-Hammond, 2011; Vescio, Ross, & Adams, 
2008). Teacher learning communities (ILCs) are examples of such PD model (Vescio et al., 2008). 
After reviewing 11 studies that reported positive effects of TLCs on teaching and learning, Vescio 
and colleagues indicate that effective TLCs focus on the significance and nature of teacher 
collaboration. Within TLCs, professional learning occurs within a collaborative culture in which 
educatots are actively engaged in collective inquiry and act on the knowledge and skills learned. 
Teacher authority—the ability of teachers to make decisions regarding the processes of theit 
learning communities—is also an important element of effective TLCs that foster changes in 
teaching cultures and mindsets. Teacher authority empowers teachers to take ownership of their 
own learning. Lastly, TLCs need to address teachers’ need for continuous learning and improvement 
in their profession (Philipsen, 2019; Vescio et al., 2008). Vescio et al. conclude their review by noting 
that participation in TLCs impacts teaching practice as teachers become more student-centered and 
focus on applying the skills and knowledge learned in their teaching. TLCs also improve teaching 
culture because the learning communities increase collaboration, empower teachers, and support 
teachers’ continuous learning needs. As a result, students benefit from TLCs as indicted by 
improved achievement scores observed in the literature. 


Darling-Hammond et al. (2017) expanded their review on PD studies beyond TLCs, and identified 
seven shared features of effective PD. First, effective PD is content-focused, job-embedded, and 
aligned with school and district priorities. Second, active learning is a key ingredient of effecttve PD 
as it engages teachers directly in the practices they are learning and is connected to teachers’ 
classrooms and students. As an example, teachers can engage in the same learning activities they are 


? The i3 grant was funded by the Office of Improvement and Innovation within the U.S. Department of Education. 


designing for their students as a form of active learning (Darling-Hammond et al., 2017). Third, 
collaboration is a common and crucial element of effective TLCs. In Darling-Hammond et al., 
collaboration includes a host of configurations—from one-on-one or small-group interactions, to 
schoolwide collaboration, to exchanges with other professionals beyond the school. The authors 
note that collaborative approaches to PD can be an effective way of promoting school change that 
extends beyond individual classrooms. Fourth, effective PD often utilizes models of effective 
practices. Examples of modeling are videos, demonstration lessons, unit or lesson plans, peer 
observations, and curriculum materials such as sample assessment and student work samples. Fifth, 
feedback and reflection are two important elements found in effective PD as these practices provide 
time for teachers to practice, receive feedback, reflect, and make changes to their practice. Sixth, 
coaching often plays an essential role in facilitating some of the strategies discussed above, such as 
modeling, collaboration, and feedback and reflection. Lastly, effective PD often requires time and 
quality implementation, which means PD that is sustained offers multiple opportunities for teachers 
to engage in learning around a single set of concepts or practices. By returning to PD settings over 
time, teachers have an opportunity to reflect, refine, and apply their understanding of materials in 
their classrooms (Darling-Hammond & McLaughlin, 2011). 


Challenges faced by rural educators in accessing evidence-based PD. Rural schools and 
districts tend to be small, and educators, who are often drawn primarily from the local population, 
have strong attachments and commitments to their communities. This environment would appeat to 
be a natural context for fostering teacher collaboration and finding opportunities to focus on issues 
and challenges of relevance to the student population (Howley & Howley, 2005). However, research 
has found that such opportunities are rare due to limited resources and geographic constraints 
(Gaumer Erickson et al., 2012; Howley & Howley, 2005; Peltola et al., 2017). It is well documented 
that rural educators, especially those who teach high needs students, often experience professional 
isolation because there ate fewer job-alike educators nearby with whom they can collaborate 
(Gaumer Erickson et al., 2012; Howley & Howley, 2005). Moreover, rural schools are less likely than 
nontrural schools to have a local professional development planning team in place, and rural teachers 
are less likely to have access to PD and are mote likely to report scheduling conflicts compared to 
nonrural teachers (Peltola et al., 2017). 


Emerging solutions for rural teachers’ PD. 'To addtess the challenges faced by rural educators, 
researchers have explored some innovative alternative approaches to delivering evidence-based PD 
to geographically isolated teachers through an online or blended learning format. Research in higher 
education has found that students in online and blended learning conditions performed statistically 
significantly better than those learning the same materials through traditional face-to-face instruction 
(Means, Toyama, Murphy, Bakia, & Jones, 2009). Online and blended learning have gained 
popularity in higher education settings because of their potential to provide learners with the 
flexibility to access the content at their own time, place, and pace (Means et al., 2009). With rapid 
development of educational technology, different applications have been developed to support 
different models of online learning and facilitate personalized learning. For instance, depending on 
the leaning objectives, some online learning platforms may use asynchronous communication tools 
(e.g., email, threaded discussion boards) to facilitate learning. Some may apply synchronous 
technologies (e.g., webinars, chat rooms, videoconferencing) to simulate face-to-face teaching 
strategies such as delivering lectures and holding meetings with a group of learners. In many cases, 
online learning incorporate multiple forms of asynchronous and synchronous online interactions to 
expand and support a community of learners. For instance, synchronous technologies can be applied 
to simulate face-to-face experiences and facilitate group discussions with groups of learners in 


different locations. Synchronous technologies can be applied to engage learners in self-reflection and 
feedback—strategies that are commonly observed in effective PD. 


Research suggests that online instruction results in greater communication and collaboration among 
class participants, and the effect is sustained beyond the course (Glomb, Midenhall, Mason, & 
Salzberg, 2009). Online communities of practice also have the potential to improve rural teacher 
retention by lessening professional isolation among new teachers (Glomb et al., 2009; Herrington, 
Herrington, Kervin, & Ferry, 2006; Knapczyk, Chapman, Rodes, & Chung, 2001). Gaumer Erickson 
and colleagues (2012) examined the effect of asynchronous online PD on secondary special 
education teachers’ practice of providing transition services for secondary students. Participants in 
the study showed increased knowledge and personal capacity to apply research-based practices, and 
reported implementation of research-based practices within their classrooms. Participants also 
reported they developed meaningful collaborative relationships with job-alike peers across the 
country. Gaumer et al. also collected data from nonrural teachers and compared them with rural 
teachers: The rural teachers, who initially rated their competency in providing transition services 
much lower, reported a greater gain at the end of the course compared to their nonrural peers. 


Blended learning is an instructional model that combines traditional face-to-face learning with online 
learning. Blended learning focuses on practices that promote personalized learning where learner- 
centered pedagogy is valued and enacted (Christensen, Horn, & Staker, 2013; Hilliard, 2015). A well- 
designed blended learning PD course can increase access, flexibility, and mobility for learners; 
generate interest in self-study for professional improvement; and empower teachers and increase 
teacher authority (Hilliard, 2015; Philipsen, 2019; Zilinskiene, Malinauskiene, & Smith, 2016). 
Research on the impact of blended learning PD is limited, but a few exploratory studies provide 
promising findings (Berger, Eylon, & Bagno, 2008; Bug, 2018; Sinclair & Owston, 2006). For 
instance, Sinclair and Owston (2006) examined the effects of two one-year blended learning PD 
courses on teachers’ attitudes, knowledge, and classroom practices. They found the programs 
increased teacher attitudes and content knowledge in the curricular areas and motivated teachers to 
transform their classroom practices. The blended learning PD also increased teacher collaboration 
and involvement at the school level, suggesting that the experience contributed to the emergence of 
communities of practice (Sinclair & Owston, 2006). Furthermore, Mirriahi and colleagues (2015) 
suggest that a well-designed blended learning course should address three areas, including resources 
(e.g., availability of course resources), activities (i.e., selection of activities for online and face-to-face 
delivery, technology to meet learners’ diverse needs and preferences, online activities to support 
learning), support (i.e., opportunity for interaction, feedback to support learning), and assessment (1.€., 
design of assessment tasks, self and peer assessment). In summary, online and blended learning may 
be a promising solution to address the PD needs of rural educators. 


Engaging principals and teachers in teacher PD. The premise of evidence-based PD is that PD 
changes teachers’ classroom beliefs and practices, which leads to improved student outcomes 
(Evans, 2014; Guskey, 2002; Philipsen, 2019; Whitworth & Chiu, 2015). To ensure that PD leads to 
teacher change as well as high-quality PD content and processes, research also highlights the 
importance of principal and teacher engagement in teacher PD (Koonce, Pijanowski, Bengtson, & 
Lasater, 2019; Philipsen, 2019; Whitworth & Chiu, 2015). Teacher engagement in teacher PD means 
that PD should be driven by teacher needs; teachers need to have the need prior to engaging in the 
actual PD. (The need can be driven by a personal need, or it can be something arising from a policy 
change or a new initiative.) One approach to ascertaining teacher engagement in teacher PD is to 
conduct a needs assessment and use data to inform PD content, processes, and designs. Koonce and 
colleagues (2019) administered a survey with 249 principals from 112 different systems which 


revealed that the majority of school principals (72%) reported that their school conducts regular 
needs assessment to guide planning for teacher PD. In these schools, teachers were involved in the 
planning process through a professional development council, and teacher choice was valued in the 
planning process. In contrast, in schools where needs assessment was not part of the PD process, 
principals generally reported a disconnect between teachers’ needs and PD provided. 


Principal engagement, on the other hand, not only plays a significant role in the planning and 
implementation of teacher PD at the school level, but also a key element that facilitates and sustains 
teacher change as a result of PD (Evans, 2014; Robinson, Lloyd, & Rowe, 2008; Whitworth & Chiu, 
2015). Research has documented multiple pathways through which leadership exerts its influence on 
student achievement. These pathways are centered around teacher attitudes, including commitment, 
trust, collective efficacy, agency, and academic optimism (Hallinger, Liu, & Piyaman, 2019; Koonce 
et al., 2019). Robinson, Lloyd, and Rowe (2008) conducted a meta-analytic review of research on 
school leadership practices that increase student achievement. They reviewed 27 studies focusing on 
five types of leadership practices: (1) establishing goals and expectations; (2) resourcing strategically; 
(3) planning, coordinating, and evaluating teaching and the curriculum; (4) promoting and 
participating in teacher learning and development; and (5) ensuring an orderly and supportive 
environment. Results of the meta-analysis revealed that leadership practices that promote and 
participate in teacher learning and development were the most effective strategy to increase student 
achievement among all other types of leadership practices. Specifically, the average effect size for 
leadership practices involving promoting and participating in teacher learning and development was 
0.84, compared to others with ES ranging from 0.27 and 0.42. Therefore, principal engagement 
should be a key element of PD programming to strengthen and sustain the impact of PD on teacher 
change in practice and student learning outcomes (Evans, 2014; Koonce et al., 2019; Whitworth & 
Chiu, 2015). 


SEED Logic Model 


Guided by the research on evidence-based PD for rural educators (Darling-Hammond et al., 2017; 
Gaumer Erickson et al., 2012; Howley & Howley, 2005; Koonce et al., 2019; Vescio et al., 2008), 
SEED took a systematic approach to addressing the PD needs of a// educators within the NW 
BOCES regions. Figure 1 shows SEED’s logic model, depicting SEED’s strategies to address the 
needs of rural educators in three components: SEED creation, SEED delivery, and SEED 
participation. 


: Short-term Long-term 
SEED Strategies Outputs Gutcomes Outcciies 
SEED Creation e SEED e Increased principal and e Improved 
° Create evidence-based PD strategies were teacher engagement in student 
driven by teacher needs and implemented teacher PD (Goal 2) achievement 
INESISE as plannedin  e Increased teacher access to (Goal 5) 
e Develop a repository of online, all SEED evidence-based PD (Goal 3) 
up-to-date, evidence-based PD ~— schools (Goal e Increased teacher application 
resources aligned with Educator —) of knowledge and skills 
Effectiveness standards learned from PD in classroom 
SEED Delivery (Goal 4) 
e Deliver PD through a dynamic e Improved student 
PD platform that enables online engagement (Goal 6) 
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Figure |. SEED Logic Model 


Creation of evidence-based PD driven by teacher needs and interest.'To engage teachers in 
PD, it was essential to ensure that PD offered would be not only driven by teachers’ interests but 
also by their needs. The activities envisioned included hiring of Innovative Coaches (ICs) to 
collaborate with content development teams (CDTs) (1.e., representatives of educators from 
districts) to identify the topics (1.e., learning strands) for PD offerings. To ascertain that the topics 
identified address both the needs and interests, data from teacher evaluation ratings would be used 
to inform areas for improvement and CDTs would receive feedback to incorporate teacher interests. 
Once the learning strands were identified, ICs would be responsible for developing and designing 
evidence-based PD. To make it likelier that the PD would improve student achievement and 
teacher practices, key features of effective PD would be incorporated in all course design and 
planning (Darling-Hammond et al., 2017; Mishkind, 2014; Vescio et al., 2008). 


Delivery of PD through a dynamic PD platform that enables online PD and blended 
learning TLCs. This strategy would strive to addtess the accessibility issue that rural educators 
often encounter. SEED would include various types of evidence-based PD, allowing for greater 
flexibility in terms of time commitment and travel requirements for rural teachers. Therefore, SEED 
PD would be delivered in two forms: online PD and blended learning TLCs. All SEED PD 
offerings would be delivered through a dynamic PD platform, allowing ICs to leverage technology 
to facilitate SEED PD in a meaningful and effective manner. Additionally, a dynamic PD platform 
in combination with blended learning TLCs would increase opportunities for NW BOCES teachers 


to collaborate across districts, and to access high quality PD resources in ways previously 
unattainable, especially for those whose teaching role is specific to high need students. 


Development of a repository of online, up-to-date, evidence-based PD resources aligned 
with Educator Effectiveness standards. NW BOCES serves more than 550 educators across 
seven school districts. To reach and serve a// teachers, the third strategy was to develop an online 
platform—Personalized Accessible Knowledge (SEED PAK)—as a central repository of searchable, 
evidence-based, and high-quality online PD resources that are available and accessible to all teachers 
24/7. SEED PAK content would be hand-picked by ICs, who would be responsible for ensuring 
content is relevant, current, and aligned with the standards specified in the teacher evaluation system 
(Educator Effectiveness standards). 


Engage principals and teachers in teacher PD. Recognizing the importance of engaging 
principals and teachers in teacher PD, principals and teachers would be encourages to utilize teacher 
evaluation results to identify appropriate resources from SEED PAK to address each individual 
teacher’s PD needs. Because SEED PAK would be structured to align with each Educator 
Effectiveness standard, principals would need to be able to easily identify the appropriate PD 
resoutces for teachers based on the areas of growth needs. Teachers could also utilize SEED PAK 
to search for topics of personal interest. 


It was hypothesized that, if all elements of SEED were implemented as planned (i.e., high fidelity; 
Goal 1), SEED would increase principal and teacher engagement in teacher PD (Goal 2); increase 
teacher access to evidence-based PD (Goal 3); and increased teacher implementation of knowledge 
and skills learned from PD in classroom (Goal 4). These changes in classroom practice would result 
in improved student engagement (Goal 5) and then student achievement (Goal 6). 


Population Served by SEED 


A total of 21 K—12 traditional public schools across seven school districts voluntarily participated in 
SEED implementation. They included nine elementary schools, three middle schools, three high 
schools, two elementary and middle schools, three middle and high schools, and one K—12 school. 
In the 2018-19 school year, one elementary school was closed, and most of its students enrolled in 
the other two SEED schools within the district. Three schools (14%) were in remote areas, and 18 
(86%) in outlying towns or cities where access to high quality PD is often a challenge for educators. 
Fifteen (71%) schools had student enrollment of less than 400 (Le., small schools), and six (29%) 
had student enrollment of 400 or more. The average percentage of minority students across SEED 
schools was 21%, ranging from 6—41% in 2016. The average percentage of FRL students was 38%, 
ranging from 17-61%. The average percentage of EL was 12%, ranging from 4-26%. Because 
SEED was a school-level intervention, the target population to be served by SEED consisted of all 
principals, assistant principals, teachers, and students. 


Table 1 provides the number of principals and assistant principals, teachers, and students across the 
21 participating sites by school year. 


Table |. Number of Principals, Assistant Principals, Teachers, and Students by 
School Year 


Full Implementation 


2015-16 


Participants (Baseline) 2016-17 PAU WYANT 2018-19 PAU EAU) 
CET) CET) CE) 
Number of principals 21 21 223 218 212 
(assistant principals) (11) (11) (10) (10) (10) 
Number of teachers 492 495 492 502 500 
(range per school) (12 - 43) (12-43) (13-44) (12-46) (13-47) 
Number of students 7,069 7180 7348 7320 7179 
(range per school) (84 - 715) (141 - 755) (139-827) (133-828) (118-839) 


a One of the participating schools has two principals. 
Source: https:/Awww.cde.state.co.us/cdereval/staffcurrent 


Evaluation Overview 


The project design involved three phases (see Table 2). Phase I focused on the development of the 
grant components. Phase I involved pilot testing of TLC components and baseline data collection. 
Phase III started in the 2016-17 school year and involved full implementation of all four SEED 
strategies. 


Table 2. SEED Project Phases 


Phase | Phase Il Phase Ill 
Spring 2015 2015-16 2016-17 | 2017-18 | 2018-19 | 2019-20 
Development Pilot testing & baseline data collection Full Implementation 


McREL International (McREL) was the third-party evaluator for the SEED project. McCREL 
conducted both formative and summative evaluation annually to understand whether and how 
SEED achieved its project goals as depicted in the logic model (see Figure 1). Formative data were 
shared with SEED staff to inform continuous program improvement; summative data were 
analyzed to monitor project progress throughout the project period. 


As shown in Table 3, six sets of evaluation questions, including a mix of process (P) and outcome 
(O) questions, were examined. These questions were organized by project goals and were answered 
using different research designs (i.e., correlational designs, quasi-experimental design [QED] using 
propensity score matching methods [PSM]). Mixed methods were conducted to gather data from 
various groups, including teacher interview (TT), teacher surveys (TS), principal interview (PI), 
principal survey (PS), IC interviews (ICI), student engagement survey (SES), project records (PR), 
and extant data (ED). This approach allowed McREL evaluators to use various data sources to gain 
a comprehensive understanding of how SEED strategies were implemented, and the extent to which 
SEED achieved its project goals. A detailed technical report of the evaluation plan, including 
evaluation design, evaluation questions, study samples, data collection methods, and data analysis, 
appears in Appendix A. 


Table 3. Evaluation Questions, Evaluation Design, and Data Collection Method by 
Project Goal 


Evaluation 


: Data 
Evaluation Questions =I Peston Collection 


>) Methods 


C QED 
Goal 1. Successfully implement and continually improve SEED in up to 30 NW BOCES schools during a 
four-year period (timeliness and continuous improvement) 


a. Are project activities occurring as planned? X PR, ICI, Pl, 

b. Is the project maintaining its planned rigor? X PS, TI, TS, 

c. What barriers and unanticipated outcomes were encountered, and what X TS 
revisions were made during implementation? 

d. What were key project successes during implementation? X 


Goal 2. Improve principal engagement in teacher professional growth and s 


a. Are principals more engaged in teacher professional growth? X Pl, PS, TI, TS 
b. Are principals able to provide more and better support to teachers through x 
SEED? 
c. Do SEED teachers, in comparison with non-SEED teachers, report a X TS 
higher level of principal engagement in teacher professional growth?* 
d. What is the relationship between school-level SEED participation rates, Xx PS, TS 


and principal and teacher reports of principal engagement in teacher PD?* 
Goal 3. Increase rural teachers’ access to and use of current best practices up-to-date content 


knowledge 

a. Are teachers utilizing SEED? Which activities are teachers using? Does X PR, TS 
use increase over time? 

._ Is PD reflective of best practices and up-to-date content knowledge? 

c. Do teachers have time and resources to access PD? 

d. Is PD offered to teachers driven by data and does it meet the needs of 
individual teachers? 

e. Do SEED teachers, in comparison with non-SEED teachers, give higher PS, TS 
ratings regarding the extent to which (1) PD is reflective of best practice 
and up-to date content knowledge; (2) PD offered is data-driven and X 
aligned with individual needs; and (3) teachers have time and resources to 
access PD?* 

f. What is the relationship between school-level SEED participation rates and PS, TS 
teachers’ ratings on the extent to which (1) PD is reflective of best practice 
and up-to date content knowledge; (2) PD offered is data-driven and X 
aligned with individual needs; and (3) teachers have time and resources to 
access PD?* 


PI, PS, TI, TS 


><| >< 


Goal 4. Support teachers in successfully implementing SEED practices and 


a. Does teachers' knowledge of best practices increase? X TS 
b. Do teachers demonstrate practice change in the classroom? X TI, TS, ED 
c. Do SEED teachers, in comparison with non-SEED teachers, (1) give higher TS, ED 
ratings on their knowledge of best practices, (2) have more opportunity to X 
collaborate with job-alike peers, and (3) have higher teacher evaluation 
scores?” 
d. What is the relationship between school-level SEED participation rates and Xx TS, ED 


(1) teachers’ ratings on their knowledge of best practices, (2) teachers’ 


Evaluation 
Design 


Data 
Evaluation Questions 0 | Collection 
P C QED i ednvere ks 


opportunity to collaborate with job-alike peers, and (3) teacher evaluation 

scores?* 

a. To what extent do SEED schools and comparison schools differ in student ED 
achievement outcomes after one year, two years, and three years of full X 
implementation? 

b. To what extent do students from SEED schools demonstrate an increase in ED 
student achievement outcomes over time?* 

c. To what extent do high-needs students from SEED schools and ED 
comparison schools differ in student achievement outcomes after one year, X 
two years, and three years of full implementation? 

d. To what extent do high-needs students from SEED schools demonstrate an ED 
increase in student achievement outcomes over time?* 

e. Do schools with higher levels of teacher participation have higher overall Xx 
student achievement?* 

a. Does student engagement increase among participating schools? X SES 

b. Do schools with higher levels of teacher participation have higher levels of X PR, TS, SES 
student engagement?* 

Note. In 2017-18 (grant Year 3), the data collection plan was strategically revised to reduce the redundancy of information 

collected over time. Specifically, principal, teacher, and IC interviews were removed from the evaluation data collection plan. 

The teacher survey was revised to include questions related to teacher perceptions of principal engagement in teacher PD 

that were originally collected through teacher interview. 

“Additional questions added in the 2017-18 reporting period. 


PR, TS, ED 


Purpose and Organization of the Report 


The main purpose of this report is to address all questions based on cumulative data collected 
throughout the project period to understand how the project strategies were implemented; to 
document the modifications made and reasons for the modifications (Goal 1); and understand the 
extent to which the project achieved the short- and long-term outcomes depicted in the logic model 
(see Figure 1, Goals 2—6). Evaluation findings are provided in the following sections organized by 
project goal. 


Evaluation Findings 


This section presents findings, by project goal, for the proposed evaluation questions. All data 
collected throughout the project period were utilized to provide a comprehensive understanding of 
the processes of project implementation and outcomes achieved throughout the implementation 


period. 


Goal |: Successfully Implement and Continually Improve SEED 
in up to 30 NW BOCES Schools During a Four-Year Period 


Four questions were examined to understand the extent to which the elements of SEED (creation, 
delivery, and participation; see Figure 1) were implemented as planned (Goal 1a) and with rigor 
(Goal 1b). Additionally, as part of the process evaluation, questions related to implementation 
challenges and successes, as well as modifications made to the implementation plan and for what 
reason, were also examined (Goals 1c and 1d). Findings from the evaluation follow. 


Goal la. The project activities were largely completed as planned. Table 4 presents a list of 
milestones established by the SEED project team during the pilot year and were used as a 
monitoring tool throughout the implementation period. By the end of the Y4 full implementation, 
the SEED project team had completed all milestones. Of those completed, < 1% (1 out of 32) were 
implemented with modification, 1% (3 out of 32) were implemented as planned but with delay, and 
1% (4 out of 32) were implemented with modification and delay. The modifications were primarily 
made to support the development of the SEED PAK platform during the development phase (stage 


1), 


Table 4. Project Implementation Milestones 


2 F 
=| o 
= (an) 
| — fl) 
i | S| F\2 
Milestone Deadline Si sis 
Scoot |e |e 
ao 2s = 
= fos 
g 5 
Initial Planning 
a. Hire part-time lead innovation coach (IC) Jan. 2015 X 
b. Extend Professional Development & Technology Integration Coordinator's contract to | Jan. 2015 . 
include SEED responsibilities 
c. Convene the NW BOCES PD Committee Jan. 2015 Xx 
d. Contract with McREL Feb. 2015 X 
e. Contract with technology service providers Apr. 2015 X 
f. Hire two ICs Apr. 2015 x 
Stage 1: Technology Development | 
a. Initiate the Platform Development Team Jan. 2015 X 
b. Conference with RANDA Solutions on work plan Jan. 2015 Xx 
c. Platform Development Team creates and refines SEED instructional model Jan. 2015 Xx 
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2 F 
=) 
- (an) 
<|/\s|is 
; e|s| 2 
Milestone Deadline 7S (ee acs 
fe ee lee 
ae 2 = 
= a 
& § 
d. Selection rubric for technology vendors created Feb. 2015 X 
e. Technology vendors selected via competitive RFP process Mar. 2015 Xx 
f._ First “sandbox” available for Platform Development Team review May 2015 X | Xx 
g. Team's feedback provided to RANDA Solutions May 2015 X 
h. Second “sandbox” for team’s review Jun. 2015 Pale 
i. Team's feedback provided to RANDA Solutions Jun. 2015 X 
j._ Final Beta SEED platform ready to launch Fall 2015 X 
k. Monthly and as-needed communication between RANDA Solutions and project Fall 2018 ae 


leadership team on technology adjustments 
Stage 2: Content Development and Phased-In Implementation 


Stage 3: Content Development and Phased-In Implementation 


a. NW BOCES PD Committee makes determination for 3 initial learning strand topics May 2015 , 
and subtopics 
b. Initiate SEED PAK development team Jun. 2015 X 
c. Innovation Coach training Jun. 2015 X 
d. Content for one learning stand is ready for Beta deployment Sept. 2015 Aailtex 
e. Initiate Content Development Teams for 3 learning strands Oct. 2015 X 
f. Content for all three initial online learning strands is ready for pilot deployment Jan. 2016 X 
g. Featured resource beta-test with SEED PAK Feb. 2016 X 
h. Extend Lead Innovation Coach's position to full time Oct. 2016 X 
i. Additional learning strand(s) determined by the PD Committee ° and new Content May 2016-2020 
Development Team(s) developed * 
j. Strategies employed to effectively roll out SEED PAK Mar. 2015-2020 | x 
k. Teachers recruited to participate in fall/spring TLC learning strands May 2016-2020 | x 


a. Completion of SEED facilitation manual Dec. 2019 Xx 
b. Training of teacher leaders and principals for continued SEED facilitation Jan. 2017-2020 | x 
c. Development of regional coaches’ network for district instructional coaches— Aug. 2015- 
; a X 
incorporating training on using SEED 2020 
d. BOCES sustainability plan is developed Jul. 2018 X 


Goal 1b. The project was mostly implemented at an adequate level and with planned rigor. 
A fidelity matrix was developed to assess the extent to which the project was being implemented as 
planned with an adequate level of implementation (see Appendix A, Table A7). Table 5 shows the 
fidelity score for each fidelity indicator throughout the project period. Key findings from the fidelity 


matrix are: 


e The SEED project team met the adequate level of implementation on all indicators under 


SEED PD Creation and SEED PD Delivery since Y1 full implementation. The level of 


3 As of 2018, the PD Committee is called the Professional Learning Advisory Group. 
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implementation on these indicators was consistent and maintained throughout the project 
period. 

e The overall level of implementation on the SEED Participation component was below the 
desired level. Specifically, the desired threshold for an adequate implementation was 3; yet, 
by the end of the implementation period, the overall level achieved was 2. As shown in 
Table B2, two indicators were identified under the SEED Participation component relating 
to SEED PAK use and SEED coutse participation. Because this component required buy-in 
from teachers and principals, it took time to achieve the desired level of implementation. 
SEED coutse participation, in particular, took three years to achieve the desired level of 
implementation. SEED PAK use, on the other hand, was on the right trajectory by the end 
of Y3 full implementation; however, the level of use declined during the final year of 
implementation. 


Table 5. Fidelity of Implementation Results 


Level of Implementation 


2017 2018 2019 
(Y1) (¥2) (Y3) 


2020 
(Y4) 


Hare [fers\ 0) es} Definition 


1. Topics for learning strands 


for PI) are selociad based Learning strands chosen based on teacher 


needs. 
on teacher needs. 
2. Evidence-based PD is 
sree (a ey [osaiis Cue PD activity has a theoretical evidence base. 3 3 3 3 


for inclusion on the SEED 
PAK. 
3. ICs and CDTs create in- 


TLCs include at least two in-person activities per 


person TLC activities. 


semester. 


4. ICs and CDTs create 
virtual TLC activities. 


TLCs have at least one hour per week of virtual 
activities. 


5. ICs format TLC delivery 
using evidence-based 
delivery standards. 


TLCs utilize the four evidence-based TLC 
standards from Vescio et al. (2008) model. 


Score for SEED PD Creation Component (threshold for adequate 


implementation = 3) 
6. ICs facilitate blended 
learning TLCs. 


TLC is hosted for each learning strand. 


Score for SEED PD Delivery Component (threshold for adequate 


implementation = 3) 

7. SEED PDis linked to 
specific teacher 
performance indicators by 
teachers or principals. 


Principal or teacher assigns at least one SEED 
PD in COPMS. 


8. Teachers complete SEED 
PD. 


implementation = 3) 


Teacher completes at least one facilitated SEED 
PD (e.g., TLC, Professional Practice Study) OR 

self-directed completion of at least 8 SEED PAK 
activities per academic year. 

Score for SEED Participation Component (threshold for adequate 


See Appendix C1 for more details about how the scores were given for each component. 


Goal 1c. SEED encountered challenges on the PD Participation components, the team 
adjusted the implementation plan as necessaty to ensure project goals were achieved. As 
reflected in the findings from the fidelity assessment, the implementation of SEED was largely 
successful in the areas of SEED PD Creation and SEED PD Delivery. The major challenge was to 
increase teacher participation in SEED courses as well as increase teacher and principal use of 
SEED PAK to support teacher PD. To achieve project goals, the project team adjusted its PD 
course offering plan by expanding SEED course offerings from TLCs to include other types of 
courses that gave teachers greater flexibility in terms of time commitment and learning modes (Le., 
online or hybrid). In its inception, SEED started with offering semester-long blended learning TLCs 
to teachers. As the project progressed, different types of SEED courses were developed and added. 
With more-diverse courses available, teachers were able to choose the types of courses that met their 
unique needs and, as a result, the desired implementation level in terms of teacher participation in 
SEED courses across SEED schools was reached by the end of Y3 full implementation. Table 6 
shows the other non-TLC types of SEED courses offered to teachers throughout the project period. 


Table 6. Other Non-TLC SEED Courses 


Non-TLC SEED Courses 
Professional Practices Studies (PPS; Began in the Spring of 2017) 


In response to post-course surveys from PLC participants and feedback from educators across the region, SEED 
decided to create PPS—a fully online and shorter course offering—that expects teachers to engage in learning 
over a six-week period and requires only 15 contact hours. With this condensed timeline, it was determined that 
the guiding learning targets would have to be more finite. Innovation Coaches identified key professional practices 
in the teacher evaluation rubric to guide the design of the learning. All elements of the original TLCs were present, 
aside from the ongoing aspect (though one could argue 15 intense hours could be considered ongoing) and 
blended. So this format differed from TLCs yet still met project goals. 
As the ICs became more informed about practices that increase student achievement, teacher leadership was 
identified as a potential growth area that had yet to be tapped. In the 2016-17 school year, SEED informally 
began to identify SEED champions, mostly around the SEED PAK, called SEED Gurus. These individuals were 
culled from content development team (CDT) members and teachers who had participated in a TLC and were 
trained to support their peers in using SEED PAK. The ICs decided to expand the program beyond just the use of 
SEED PAK, so every school in the NW BOCES had a designated and informed teacher leader, trained by SEED. 
The SEED Facilitators were identified by building principals and were paid a $2,000 stipend for their services. 
Facilitator meetings were much like TLC meetings. They were based on best practice in teacher leadership, 
collaborative, allowed opportunities for reflection, and were designed to get the heart of our project—enhance 
student learning and achievement. In addition to these quarterly, all-day, in-person meetings, these leaders were 
tasked with creating learning opportunities for their peers based on the needs of their staff and students. 
Book Studies (Began in the Fall of 2018) 
In the second year of the SEED grant, SEED engaged in a design thinking process to identify how to better meet 
the needs and wants of teachers. After examining the characteristics of SEED participants, it was clear that SEED 
had a higher level of participation from elementary than secondary teachers. Through the design thinking process, 
the ICs found that secondary teachers were most interested in PD that was targeted to their content area. Based 
on that observation, and paired with student achievement data, the first book study was based on Jo Boaler’s 
Mathematical Mindset. The book study was a 15-hour commitment, and featured an in-person meeting to begin 
the learning, coaching, implementation, and reflection expectations. All of the key elements of evidence-based 
TLCs and best practices in adult learning were applied to the course design. It was a huge success and attracted 
teachers who had yet to participate in SEED programming. As time went on book studies became fully online, and 
varied in time commitment, yet were still effective in meeting project goals and purpose. 


SEED Independent Study (Began in the fall of 2018) 


Non-TLC SEED Courses 


This course was created to allow teachers to drive the content of their learning and to bring more users to SEED 
PAK. Shorter than all other SEED courses at a 7.5-hour time commitment, teachers met with a coach to clarify 
their goals for their learning and then were expected to complete SEED PAK PD Market resources based on their 
goals. Participants were also tasked with implementing new practices in their classroom based on the learning, all 
while being supported by a coach. Many elements of TLCs were present, just in a way that met a teacher's desire 
for very personalized and targeted learning. 


SEED also offers several other types of licensure-based courses/programs to support teachers within NW 
BOCES, including alt-licensure, induction program, and EL learning modules. 


Adoption of SEED PAK was a challenge throughout the life of the grant. The project team 
consistently strategized and identified new ways to increase use. One way to achieve that was to 
encourage SEED PAK use through the SEED Facilitators program and SEED Independent 
studies. After adding these two types of course offerings, there was a noticeable level of increase in 
SEED PAK usage starting in Y2 (13% of teachers in SEED completed at least one SEED 
resource), and that level was repeated in Y3. However, the usage level decreased again in Y4. Before 
concluding the project, the project team reflected on the challenges with implementation of SEED 
PAK and noted, 


Although SEED PAK was well-designed and stocked with excellent resources 
connecting to the teacher evaluation rubric, many teachers were just not interested in 
using it. The design thinking process led us to the conclusion that SEED PAK simply 
lacked the human connection that teachers craved when learning. Although the level 
of SEED PAK use was not what we originally hoped for, some teachers find value in 
it. 


Goal 1d. The project was successful in terms of developing and delivering evidence-based 
and up-do-date PD content. Additionally, with persistence in consistently strategizing and 
identifying new strategies to engage teachers in SEED PD, the project was able to reach the 
designed level of implementation on SEED course offerings atter three years of 
implementation. Since project inception, SEED had focused on creating a PD system to support 
the needs of educators in the rural areas served by NW BOCES. Throughout the years, SEED had 
created evidence-based PD courses driven by teachers’ needs and interests, and developed a 
repository of online, up-to-date, evidence-based PD resources (i.e., SEED PAK) for all educators 
within the schools that NW BOCES serves. To ensure that the PD courses met the needs of 
educators within the NW BOCES region, SEED formed a NW BOCES PD Committee (then the 
Professional Learning Advisory Group) to determine the topics for SEED offerings based on 
various data sources, including examining teacher evaluation results, results of teacher and principal 
surveys, SEED Facilitators’ meeting notes, and RANDA self-assessment data. After the topics were 
defined, content development teams (CDTs) of teachers reviewed draft course syllabi, activities, and 
resoutces, and provided feedback that the SEED coaches then incorporated into the course design. 
With regard to the development and direction for the new learning strands, the process changed 
slightly over time. During the early stage of the project, each TLC course was considered a learning 
strand. Starting with Y3 full implementation, the SEED course offerings were intended to be 
cohesive and to provide teachers with multiple options, in terms of time requirement and learning 
mode (i.e., a combination of face-to-face and online learning), to build on their growth within the 
focus strand. 
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Table 7 presents the comprehensive list of SEED courses throughout the project period as well as 
number of participants over time. Overall, the enrollment number and completion rate increased 
annually (by 14%, 63%, and 117% from Y1 to Y2, Y2 to Y3, and Y3 to Y4, respectively). The 
completion rate also increased annually, from 87% in Y1 to 100% by the end of the implementation 


period (Y4). 
Table 7. Number of Enrollments and Completion Rates of SEED Course Offerings 
Throughout the Project Period 


Completion 
Rate 


# Completed 2 


Facilitated SEED PD # Recruited 


2016-17 School Year 
TLC 


Critical thinking & problem solving 8 8 100% 
Developing classrooms of inquiry 10 10 100% 
Effective planning for engaging instruction 7 7 100% 
Engaging all students in learning 18 14 77.8% 
Enhancing classroom instruction through technology 13 9 69.2% 
integration 

Increasing student ownership of learning 22 ze 100% 
PPS 

Handing thinking over to the students 12 10 83.3% 
Others 

Induction program 13 10 76.9% 
Alt-licensure program 4 3 75% 

Total Number of Unique Teachers? ¢ 97 84 86.6% 


2017-18 School Year 


TLC 

Connecting key ideas across content areas 11 10 90.9% 
Instructional design to maximize student learning 6 5 83.3% 
Increasing student ownership of learning 25 25 100% 
Students as partners in creating a positive classroom 9 9 400% 
environment 

Using developmental and cognitive science to enhance 44 44 400% 
instruction 

PPS 

Teaching for cognitive engagement 8 7 87.5% 
Using technology for powerful pedagogy 13 8 61.5% 
Others 

Induction program 16 13 81.3% 
Alt-licensure program 12 11 91.7% 
SEED Facilitator 23 23 100% 
Total Number of Unique Teachers®° 111 101 91.0% 


2018-19 School Year 


TLC 

Access and participation in the thinking classroom 23 23 100% 
Connecting key ideas across content areas 8 7 87.5% 
Empowering student thinking 22 21 95.5% 


PPS 


Completion 


Facilitated SEED PD # Recruited # Completed @ Rate 

Student engaged revision 9 8 88.9% 
Performance-based assessment 7 7 100% 
Book Study 

Facilitating teacher teams 14 14 100% 
Mathematical Mindsets book study 29 23 79.3% 
Supporting beginning teachers 8 6 75% 

Independent Study 

SEED independent study 46 44 95.7% 
Others 

Alt-licensure program 11 11 100% 
Architecture of authentic & relevant learning 13 10 76.9% 
EL modules 37 37 100% 
Induction program 21 19 90.5% 
SEED Facilitator 23 23 100% 
Total Number of Unique Teachers? ¢ 181 172 95% 


2019-20 School Year 


TLC 

Creating a positive classroom 11 10 90.9% 
Enhancing social emotional learning with developmental 33 33 

and cognitive science 100% 
Student collaboration 26 26 100% 
PPS 

Critical foundations of formative assessment 10 10 100% 
Book Study 

Teaching reading in the content area 9 9 100% 
Independent Study 

SEED Independent Study 36 35 97.2% 
Others 

Alt-license program 11 10 90.9% 
EL Modules 238 231 97.1% 
Induction Program 15 14 93.3% 
SEED Facilitator 21 21 100% 
Total Number of Unique Teachers? ¢ 392 392 100% 


a Data were provided by the SEED project team. Completion was defined as completion of at least 80% of the course 


requirements. 


> The total number of unique teachers was smaller than the sum of the number of teachers recruited for and completing each 
PD because some teachers participated in more than one SEED PD offering. 


¢ Teachers who completed at least one of the SEED courses were counted as completers. 


Other noteworthy successes in SEED implementation were: 


e SEED Facilitator Program 
devel her leaders, 'T | feel more empowered as a teacher leader—! came to the 
eyelnps aCe eagee hD conclusion that as a teacher leader it is my role to get 
support developing teacher leaders, | jinyojyed more to make our teaching better as a collective. 


oe a ee aes . Segoe f th | think it was helpful to do the concept map and really put 
ce te ace ee ae deep thought into what a teacher leader is. | always 


SEED facilitator meetings. enjoy getting opportunities to reflect when | am away from 
Feedback received from teachers | iffetglassroom! —SEED Facilitators 
was positive and hopeful. 


e SEED PAK usage increased noticeably in the Y2 and Y3 full implementation 
years. The growth in SEED PAK participation and completion of resources was a 
success after rolling out the SEED Facilitator program and SEED independent 
studies. Teachers who participated in the independent studies used SEED PAK 
resoutces as their supporting learning materials, which directly impacted the number 
of teachers utilizing SEED PAK resources. Additionally, SEED Facilitators were 
encouraged to use SEED PAK resources to engage teachers in their building to 
access SEED PD. 


e IC support services were of high quality. ICs played a key role in designing, delivering, 
and implementing SEED courses. When asked about the quality of ICs, teachers spoke 
highly of the services they received. Figure 2 shows the average ratings over time. Results of 
latent growth curve (LGC) modeling revealed that the average rating increased significantly 
over time (6 = 0.17, SE = 0.05, p < 0.001). 

Items measuring IC quality: 


IC Quality 
e My IC provided information and 


resources | needed to support 
my learning needs and 
professional goals. 


e My |IC was available to answer 
questions. 


e My IC was knowledgeable 
about PD topics. 


2017 (Y1 Full 2018 (Y2 Full 2019 (Y3 Full 2020 (Y4 Full 
Implementation) Implementation)Implementation) Implemenation) 


e Overall, | was satisfied with the 
quality of the coaching and 

Item scale—Each item was rated on a 6-point Likert-type scale, with the responses support | received. 

ranging from strongly disagree (1) to strongly agree (6). 


Figure 2. Teacher Perception of IC Quality Over Time 


17 


Goal 2. Increase Principal Engagement in Teacher Professional 
Growth and Support Throughout the Project Period 


SEED’s second goal was to increase principal engagement in teacher professional growth. Four 
questions were examined: Do principals feel more engaged in teacher professional growth (Goal 2a)? Are 
principals able to provide more and better support to teachers (Goal 2b)? Do SEED teachers, in comparison with 
non-SEED teachers, report a higher level of principal engagement in teacher professional growth? (Goal 2c)? What is 
the relationship between school-level SEED participation rates, and principal and teacher reports of principal 
engagement in teacher PD? (Goal 2d)? Data were collected from both principals and teachers through 
online surveys annually. To answer Goals 2a and 2b, a series of latent growth curve (LGC) models 
was conducted to examine the rate of change on each of the measured constructs over time. School- 
level means were used in the analyses. To answer Goal 2c, multilevel level modeling was conducted 
to examine the difference between SEED and non-SEED teachers while accounting for the nesting 
structure of the data. To answer Goal 2d, linear regression analysis was conducted to examine the 
difference between High SEED Participation (HP) schools and Low SEED participation (LP) 
schools.* Findings are summarized as follows. 


Goal 2a. Both principals and teachers within SEED schools reported an increased level of 
ptincipal engagement in teacher professional growth over time. Figure 3 shows the average 
level of principal engagement in teacher professional growth over time based on principal report. 
Results of LGC revealed that, within SEED schools, principal engagement in teacher professional 
growth increased significantly over time (6 = 0.15, SE = 0.03, p < 0.001). 


Principal Engagement in Teacher Professional Growth 


3.99 batty : 433 


2015 (Pilot 2016 (Pilot Follow- 2017 (Y1 Full 2018 (Y2 Full 2019 (Y3 Full 2020 (Y4 Full 
Baseline) ivy) implementation) Implementation) Implementation) Implementation) 


Item scale—Each item was rated on a 6-point Likert-type scale, with the responses ranging from strongly disagree (1) to strongly agree (6). 
Items measuring principal engagement in teacher professional growth (principal survey): 

e —_Inthis school, follow-up was provided from PD. 

e The PD my school offered was evaluated and results were communicated to teachers. 

e Teachers were encouraged to reflect on their own practice. 

e The current teacher evaluation systems provide information that allows me to be supportive of teacher PD. 

e —_|understand what PD support teachers need. 


e During conversations with teachers regarding their professional growth (i.e., improvement on the teacher evaluation rubric and in 
knowledge and implementation of best practices), | feel prepared to be supportive. 


e Overall, | was satisfied with the quality of the coaching and support | received. 


Figure 3. Principal Engagement in Teacher Professional Growth Over Time 
(Principal Survey) 


4 Definitions of HP and LP schools are provided the Data Analysis Plan section in Appendix A. 
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Figure 4 shows the average level of principal engagement in teacher professional growth based on 
teacher report over time. Results of LGC revealed that, within SEED schools, teacher perception of 
principal engagement in teacher professional growth increased significantly over time (6 = 0.19, SE 
= 0.04, p < 0.001). 


Principal Engagement in Teacher Professional Growth 


2018 (Y2 Full Implementation) 2019 (Y3 Full Implementation) 2020 (Y4 Full Implementation) 


Item scale—Each item was rated on a 6-point Likert-type scale, with the responses ranging from strongly disagree (1) to strongly agree (6). 
Items measuring principal engagement in teacher professional growth (teacher survey): 
e My principals or supervisor recommended and planned PD based on needs identified through the teacher evaluation process. 


e The current teacher evaluation systems in place provide information that allows my principal or supervisor to be supportive of 
teacher PD. 


e My principal or supervisor understands what PD support teachers need. 


e During conversations with my principal or supervisor regarding my professional growth (i.e., improvement on the teacher 
evaluation rubric and in knowledge and implementation of best practices), my principal is supportive. 


Figure 4. Principal Engagement in Teacher Professional Growth Over Time 
(Teacher Survey) 


Goal 2b. Both principals and teachers reported that SEED helped principals to provide 
more and better support for teacher professional growth over time. Figures 5 and 6 show the 
average level of SEED utility to support teacher professional growth over time based on principal 
and teacher report, respectively. Results of LGC revealed that, within SEED schools, principal and 
teacher report of SEED utility to support teacher professional growth increased significantly over 
time (principal report: 6 = 0.16, SE = 0.05, p = 0.003; teacher report: 6 = 0.27, SE = 0.05, p < 
0.001). 


19 


—— SEED Utility to Support Teacher PD 


2016 (Baseline) 2017 (Y1 Full 2018 (Y2 Full 2019 (Y3 Full 2020 (Y4 Full 
Tanve)cvanteraneciucey ay) Tanto) Kevanteyareciuleyay) aave)(edaatsvaieclateyay) ifanto)(cYantelareciuleyay) 


Item scale—Each item was rated on a 6-point Likert-type scale, with the responses ranging from strongly disagree (1) to strongly agree (6). 
Items measuring SEED utility to support teachers (principal survey): 

e SEED content is useful for teachers’ PD. 

e The SEED system has PD that is relevant to teachers’ needs as identified by the state evaluation rubric (RANDA). 

e SEED PD enabled me to support teachers’ professional growth. 

e The integration of the SEED PAK and RANDA is easy to use. 

e The integration of the SEED PAK and RANDA allowed me to better track teacher professional growth plans. 

e The integration of the SEED PAK and RANDA helped facilitate conversations with teachers about professional growth. 


Figure 5. SEED Utility to Support Teacher PD Over Time (Principal Survey) 


——= SEED Utility to Support Teacher PD 


2016 (Baseline) 2017 (Y1 Full 2018 (Y2 Full 2019 (Y3 Full 2020 (Y4 Full 
Tanve)(cvantevaneciuley ay) Tanto) Kevantevareciulelay) aave)(edaatsyaieclateyay) ifaave)(cYanteyareciulolay} 


Item scale—Each item was rated on a 6-point Likert-type scale, with the responses ranging from strongly disagree (1) to strongly agree (6). 
Items measuring SEED utility to support teachers (principal survey): 

e SEED content is useful for teachers’ PD. 

e The SEED PD is relevant to my needs. 

e The integration of the SEED PAK and RANDA is easy to use. 

e The integration of the SEED PAK and RANDA allowed me to better track teacher professional growth plans. 

e The integration of the SEED PAK and RANDA helped facilitate conversations with teachers about professional growth. 


Figure 6. SEED Utility to Support Teacher PD Over Time (Teacher Survey) 


Goal 2c. SEED and non-SEED teachers reported the same level of principal engagement in 
their professional growth. When comparing the ratings between SEED and non-SEED teachers 
using Y3 data, there was no difference in terms of teachers’ ratings on the level of principal 
engagement (8 = 0.11, SE = 0.12, p = 0.343). 
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Goal 2d. HP and LP schools’ did not differ in terms of principal engagement in teacher 
ptotessional growth. Linear regression analyses revealed that HP schools and LP schools did not 
differ on teacher or principal report of principal engagement (teacher report: 6 = 0.00, SE = 0.13, p 
= 0.999; principal report: 6 = -0.27, SE = 0.17, p = 0.137). 


Findings from Goals 2c and 2d suggest that all teachers might have benefited from SEED regardless 
of SEED participation status in the area of increased principal engagement in teacher professional 
growth. However, due to the nature of the correlational design, causal inference cannot be made. 


Goal 3. Increase Rural Teachers’ Access to and Use of Current 
Best Practices and Up-to-Date Content Knowledge 


SEED was designed to increase rural teachers’ access to and use of up-to-date, evidence-based 
practices (Goal 3). Six questions were addressed: Ave teachers utilizing SEED? Which activities are teachers 
using? Does use increase over time (Goal 3a)? Is PD reflective of best practices and up-to-date content knowledge 
(Goal 3b)? Do teachers have time and resources to access PD (Goal 3c)? Is PD offered to teachers driven by data and 
does it meet the needs of individual teachers (Goal 3d)? Do SEED teachers, in comparison with non-SEED 
teachers, give higher ratings regarding the extent to which (1) PD ts reflective of best practice and up-to date content 
knowledge; (2) PD offered is data-driven and aligned with individual needs; and (3) teachers have time and resources 
to access PD (Goal 3¢)? What is the relationship between school-level SEED participation rates and teachers’ ratings 
on the extent to which (1) PD is reflective of best practice and up-to date content knowledge; (2) PD offered is data- 
driven and aligned with individual needs; and (3) teachers have time and resources to access PD (Goal 3f)? 


For Goal 3a, the SEED participation data collected by the project team throughout the project 
period were analyzed. For the remaining questions, data collected from the teacher and principal 
surveys throughout the project period were analyzed. Specifically, to answer Goals 3b, 3c, and 3d, a 
series of LGC was conducted to examine the rate of change on each of the measured constructs 
over time. School-level means were used in the analyses. To answer Goal 3e, multilevel level 
modeling was conducted to examine the difference between SEED and non-SEED teachers while 
accounting for the nesting structure of the data. To answer Goal 3f, linear regression analysis was 
conducted to examine the difference between HP and LP schools on the outcomes of interest. 
Findings are summarized in the following sections. 


Goal 3a. Teachers participated in a wide variety of SEED activities, including participating 
in SEED courses, using SEED PAK resources, and reading SEED blogs. The overall level 
of participation in SEED courses increased significantly over time. Yet, the level of 
patticipation in SEED PAK and reading SEED blogs remained low throughout the project 
petiod. Throughout the project period, the SEED project team continued identifying new strategies 
to increase teacher participation. Over the years, the project team expanded SEED coutse offerings 
from one-semester long TLCs to other options that required less time commitment and had greater 
flexibility in terms of participation modes (i.e., online or hybrid) without compromising quality. 
Additionally, to reach as many teachers as possible, the project team added writing SEED blogs in 
their implementation plan as a means to reach a wider audience across the schools that SEED 
serves. Teacher participation in each type of activities and the quality of the activities are 
summarized as follows. 


> See Data Analysis Plan in Appendix A for more detail information regarding the definition of HP schools and LP 
schools. 
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e SEED coutse participation and course quality. Overall, SEED course offerings included 
TLCs, PPS, independent study, book study, and others (e.g., EL modules, Alt-licensure, 
Induction, SEED Facilitators). Table 8 shows the number of unique teachers who 
participated in various SEED course offerings throughout the full implementation years. 
Overall, there were increases of 3%, 14%, and 42% from Y1 to Y2, from Y2 to Y3, and 
from Y3 to Y4, respectively. The completion rate also increased annually—by the end of Y4 
full implementation, 100% of SEED participants had completed at least one SEED course 
in which they were enrolled. Data on teacher perception of the quality of SEED courses 
were collected starting with 2019 (Y3 full implementation) via the teacher survey. 


Table 8. SEED Courses Enrollment and Completion Rates Over Time 


~ # Participation Completion 

Facilitated SEED PD # FTE Teachers # Enrolled Completed Rate Rate 
2016-17 School Year 495 97 84 19.6% 86.6% 
(Y1 full implementation) 

2017-18 School Year 9 0, 

(Y2 full implementation) 492 m1 101 22.6% 91% 
2018-19 School Year ‘ : 

(Y3 full implementation) 502 181 172 36.1% 95% 
2019-20 School Year . ; 

(Y4 full implementation) 500 392 392 78.4% 100% 


Note. Data were provided by the SEED project team. Completion was defined as a completion of at least 80% of the course 
requirements. Teachers who completed at least one SEED course were counted as completers. 


When asked about the quality of SEED courses, teachers, overall, gave high ratings on the 
quality of courses in which they participated through the Y3 and Y4 online survey. As shown 
in Figure 11, the average rating increased over time, and the rate of increase was significant 
from Y3 to Y4 (6 = 1.94, SE = 0.22, p < 0.001). 


SEED Course Quality Items measuring the quality of SEED courses: 


e SEED courses were a good use of my time. 

e SEED courses have useful activities and 
content. 

e SEED courses increased my knowledge of best 
practices. 


2019 (Y2 Full Implementation) 2020 (Y4 Full Implementation) 


Item scale—Each item was rated on a 6-point Likert-type scale, with the responses 
ranging from strongly disagree (1) to strongly agree (6). 


Figure 7. Quality of SEED Courses Over Time (Teacher Survey) 


e SEED PAK usage and quality. As shown in Table 9, the number of teachers who 
completed at least one SEED PAK resource as well as the number of SEED PAK resources 
completed by teachers increased noticeably from Y1 to Y2 (11 percentage-point increase), 
and the usage level was maintained in Y3. However, the usage level dropped significantly in 
Y4 (8 percentage-point decrease). As discussed earlier (see Goal 1 findings), the increase in 
SEED PAK use in Years 2 and 3 may be the result of the project team’s effort to promote 
use through SEED courses (.e., independent study, SEED Facilitator program). However, 
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the low level of SEED PAK use overall may be due to lack of human connection and 
motivation that teachers desired when learning. 


Data on teacher perception of the quality of SEED PAK were collected via teacher survey. 
When asked about the quality of SEED PAK from teachers who have used it, teachers, on 
average, gave high ratings. As shown in Figure 8, the average rating increased over time, and 
the rate of increase was significant (6 = 0.23, SE = 0.05, p < 0.001). 


Table 9. SEED PAK Usage by Year 


Year Steen hee i aie one ion ion oie 
SEED PAK Resource Completed 
ee c Ca: 
(2 ful mplementaton) 492 z ites soll 
‘umpc | = | em | 
eee e : ‘eM 


Note. Data source is IC Participation Logs—SEED PAK usage data. 


Data on teacher perception of the quality of SEED PAK were collected via teacher survey. 
When asked about the quality of SEED PAK from teachers who have used it, teachers, on 
average, gave high ratings. As shown in Figure 8, the average rating increased over time, and 
the rate of increase was significant (6 = 0.23, SE = 0.05, p < 0.001). 


SEED PAK Quality Items measuring SEED PAK quality: 


e SEED PAK activities were a good 
use of my time. 

e SEED PAK has useful activities 
and content. 

e SEED PAK increased my 
knowledge of best practices. 


2017 (Y1 Full 2018 (Y2 Full 2019 (Y3 Full 2020 (Y4 Full 
Tanvo)(cYantevaneciuloyap ileal) (etaatevaiecua(eyap im lanto)(ciaatcvareciacolay mu laale)(vaelslarclaveye)] 


Item scale—Each item was rated on a 6-point Likert-type scale, with the responses ranging from 
strongly disagree (1) to strongly agree (6). 


Figure 8. Quality of SEED PAK Resources Over Time (Teacher Survey) 
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e Read the SEED blog and quality of the SEED blog. Starting in Y2, The project team 
decided to start writing blog posts and post them on the SEED PAK platform to reach a 
wider audience. As shown in Table 10, the number and percentage of teachers who read 
SEED blogs decreased over time. 


Table 10. Number of Teachers Who Read SEED Blogs by Year 


Number of Teachers _ Percentage of Teachers Who 


He #FTE Teachers Who Read SEED Blogs Read SEED Blog 
(2 ul mplementtion #02 7 aie 
onc, |= | on 
(4 ful impementaton 500 ‘i a 


Note. Data source is IC Participation Logs—SEED PAK usage data. 


Data on teacher perception of the quality of SEED blogs were collected via teacher survey. 
When asked about the quality of SEED blogs from teachers who have read it, teachers, on 
average, gave high ratings. As shown in Figure 9, the average rating increased over time, and 
the rate of increase was significant (6 = 0.29, SE = 0.07, p < 0.001). 


Items measuring SEED PAK 

quality: 

e Reading the SEED blogs is 
a good use of my time. 

e The SEED blog has useful 
activities and content. 

e The SEED blog increased 
my knowledge of best 
practices. 


SEED Blogs Quality 


2018 (Y2 Full 2019 (Y3 Full 2020 (Y4 Full 
ifaave)(cYanteyareciulelay) Tante)tcYantevaneciuleyay) Implemenation) 


Item scale—Each item was rated on a 6-point Likert-type scale, with the responses ranging from 
strongly disagree (1) to strongly agree (6). 


Figure 9. Quality of SEED Blogs Over Time (Teacher Survey) 


Goal 3b. Within SEED schools, PD is evidence-based and has up-to-date content. Both 
ptincipals and teachers provided increased ratings on both constructs over time. Figure 10 
shows the extent to which principals perceived PD offered to teachers to be evidence-based. Results 
of LGC revealed that, within SEED schools, principals’ ratings of evidence-based PD increased 
significantly over time (6 = 0.14, SE = 0.03, p < 0.001). Yet it should be noted that, within SEED 
schools, teachers have access to both SEED and non-SEED PD. Therefore, the trend shown in 
Figure 7 represent principals’ perception of all PD (SEED and non-SEED) available to all teachers 
within SEED schools. 
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Evidence-based PD 


2015 (Pilot 2016 (Pilot Follow- 2017 (Y1 Full 2018 (Y2 Full 2019 (Y3 Full 2020 (Y4 Full 
Baseline) Up) implementation) Implementation) Implementation) Implementation) 


Item scale—Each item was rated on a 6-point Likert-type scale, with the responses ranging from strongly disagree (1) to strongly agree (6). 
Items measuring evidence-based PD (principal survey): 


e The PD my school offered enhanced teachers’ ability to implement instructional strategies that meet diverse student learning 
needs. 


e The PD my school offered provided ongoing opportunities for teachers to work with colleagues to refine teaching practices. 
e The PD my school offered enhanced teachers’ abilities to improve student learning. 

e The professional learning opportunities my school offered were aligned with the school’s improvement plan. 

e The PD my school offered deepened teachers’ content knowledge. 

e _Myschool's PD offerings were data-driven. 

e | recommended and planned PD based on needs identified through the teacher evaluation process. 


Figure 10. Evidence-based PD Over Time (Principal Survey) 


Figure 11 shows teachers’ perceptions of up-to-date (single item) and evidence-based PD content 
(single item) over time. Results of LGC revealed that, within SEED schools, teachers’ ratings on the 
extent to which PD content is up-to-date and evidence-based increased significantly (up-to-date PD: 
6 = 0.15, SE = 0.02, p < 0.001; evidence-based PD: f = 0.17, SE = 0.02, p < 0.001). 


112016 (Baseline) 2017 (Y1 Full Implementation) 0 2018 (Y2 Full Implementation) 


2019 (Y3 Full Implementation) [ 2020 (Y4 Full Implementation) 


5.22 5.21 
a.79| |4.94 4.91 
i : : i al 


PD Content is Up-to-Date PD Content is Evidence-based 


Note. Each item was rated on a 6-point Likert-type scale, with the responses ranging from strongly disagree (1) to strongly agree (6). Amean 
score was calculated to present the average rating. 


Figure ||. Up-to-Date and Evidence-based PD Content Over Time (Teacher 
Survey) 


Goal 3c. Within SEED schools, there was an overall increase in terms of school-provided 
time and resources to access PD based on both teacher and principal reports. Figure 12 
shows principals’ and teachers’ perceptions, respectively, of time and resources to access PD over 
time. Results of LGC revealed that, within SEED schools, both principals’ and teachers’ perceptions 
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of the extent to which school offered time and resources to access PD increased significantly 
(principal report: 6 = 0.19, SE = 0.05, p < 0.001; teacher report: 6 = 0.30, SE = 0.04, p < 0.001). 


12015 (Pilot Baseline) 2016 (Pilot Follow-Up) 12017 (Y1 Full Implementation) 


012018 (Y2 Full Implementation) (2019 (Y3 Full Implementation) 1) 2020 (Y4 Full Implementation) 


= taal (94 
3 g6| |4.03 oy ee li 


Principal Report Teacher Report 


Item scale—Each item was rated on a 6-point Likert-type scale, with the responses ranging from strongly disagree (1) to strongly agree (6). 
Items measuring school-provided time and resources to access PD: 

e —_ Anappropriate amount of time was provided for PD at my school. 

e Sufficient resources were available for PD in my school. 


Figure 12. School-Provided Time and Resources to Access PD 


Goal 3d. Within SEED schools, there was an overall increase in principal and teacher report 
of school-oftered individualized teacher PD. Additionally, teachers’ ratings on the extent to 
which PD offered by the school was driven by data increased over time. Figure 13 shows 
principal and teacher perception of school-offered individualized teacher PD (single item), 
respectively, over time. Results of LGC revealed that, within SEED schools, both principals’ and 
teachers’ ratings on school-offered individualized teacher PD increased significantly over time 
(principal report: 6 = 0.17, SE = 0.05, p < 0.001; teacher report: 6 = 0.32, SE = 0.08, p < 0.001). 


12015 (Pilot Baseline) 2016 (Pilot Follow-Up) 02017 (Y1 Full Implementation) 
012018 (Y2 Full Implementation) 4 2019 (Y3 Full Implementation) 0 2020 (Y4 Full Implementation) 


rq 4.72 4.79} 44.89. 
al ii : 


Principal Report Teacher Report 


Note. Each item was rated on a 6-point Likert-type scale, with the responses ranging from strongly disagree (1) to strongly agree (6). A mean 
score was calculated to present the average rating. 


Figure 13. My School Offered Differentiated PD Over Time 


Figure 14 shows the average rating of teacher perception of the extent to which the PD offered is 
driven by data over time. Results of LGC revealed that, within SEED schools, teachers’ ratings of 
data-driven PD increased significantly over time (6 = 0.23, SE = 0.03, p < 0.001). 
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Data-driven PD 


378 4.04 2 


2016 (Baseline) 2017 (Y1 Full 2018 (Y2 Full 2019 (Y3 Full 2020 (Y4 Full 
faave)(cYantevareciucelay) ifaavol(cYantelaieciuceyay) ifanve)Kevantevareciuceyay) Tanvo)(cYantevareiucelay 


Item scale—Each item was rated on a 6-point Likert-type scale, with the responses ranging from strongly disagree (1) to strongly 
agree (6). 

Items measuring data-driven PD: 

e The professional learning opportunities | was offered were aligned with my school’s improvement plan. 

e The PD | was offered was data-driven. 

e Teachers were encouraged to reflect on their own practice. 

e School leadership recommended and planned PD based on needs identified through the teacher evaluation process. 

e —_Inmy school, follow-up was provided from PD. 

e The PD my school offered was evaluated and results were communicated to teachers. 


Figure 14. Data-driven PD Over Time (Teacher Survey) 


Goal 3e. SEED teachers reported more favorable outcomes compared to non-SEED 
teachers in terms of constructs related to access to and use of up-to-date and evidence- 
based PD. When comparing the ratings of SEED and non-SEED teachers, SEED teachers gave 
higher ratings on the following constructs: 


e Up-to-date PD (6 = 0.33, SE = 0.13, p < 0.001) 

e Evidence-based PD (6 = 0.31, SE = 0.11, p = 0.007) 

e School-provided time and resources to access PD (8 = 0.67, SE = 0.13, p < 0.001) 
e Data-driven PD (6 = 0.60, SE = 0.10, p < 0.001). 


Due to the nature of the correlational design (non-matched groups), causal inference cannot be 
made. Yet, these findings altogether are encouraging as they suggest that SEED participation was 
associated with teachers’ ratings on the extent to which the PD they received was up-to-date, 
evidence-based, and data-driven, as well as whether schools offer time and resources to access PD. 


Goal 3f. HP schools, on average, had some favorable outcomes compared to LP schools in 
terms of constructs related to access to and use of up-to-date and evidence-based PD. When 
compating the school-level ratings between HP and LP schools, HP schools had higher ratings on 
the following constructs: 


e Up-to-date PD (6 = 0.28, SE = 0.12, p = 0.027) 
e Data-driven PD (6 = 0.33, SE = 0.13, p = 0.023) 
On the other hand, HP and LP schools did not differ on the following constructs: 
e Evidence-based PD (6 = 0.22, SE = 0.14, p = 0.138) 
e School-offered time and resources to access PD (6 = 0.30, SE = 0.16, p < 0.087) 


Due to the nature of the correlational design (non-matched groups), causal inference cannot be 
made. Yet, these findings altogether are encouraging as they suggest that SEED participation at the 
teacher and school levels was mostly associated with teachers’ access to and use of up-to-date PD, 
and the PD provided was driven by data based on individual needs. 
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Goal 4. Support Teachers in Successfully Implementing SEED 
Practices and Content 


Goal 4 of SEED was to support teachers’ implementation of knowledge and practices learned from 
SEED in the classroom. Four questions were addressed: Do teachers’ knowledge of best practices increase 
(Goal 4a)? Do teachers demonstrate practice change in the classroom (Goal 4b)? Do SEED teachers, in comparison 
with non-SEED teachers, give higher ratings on their knowledge of best practices, (2) have more opportunity to 
collaborate with job-alike peers, and (3) have higher teacher evaluation scores (goal 4c)? What is the relationship 
between school-level SEED participation rates and (1) teachers’ ratings on their knowledge of best practices, (2) 
teachers’ opportunity to collaborate with job-alike peers, and (3) teacher evaluation scores (Goal 4d)? 


Data collected from the teacher and principal surveys throughout the project period were analyzed. 
To answer Goals 4a and 4b, a series of LGC was conducted to examine the rate of change on each 
of the measured constructs over time. School-level means were used in the analyses. To answer Goal 
4c, multilevel level modeling was conducted to examine the difference between SEED and non- 
SEED teachers while accounting for the nesting structure of the data. To answer Goal 4d, linear 
regression analysis was conducted to examine the difference between HP and LP schools on the 
outcomes of interest. Findings are summarized in the following sections. 


Goal 4a. PD increased teacher report of knowledge of best practices in terms of instruction 
and learning increased over time. This goal was measured by the extent to which PD increased 
teachers’ knowledge of best practices—instructional practice-focused PD and student learning- 
focused PD. Figure 15 shows the average rating of teacher perception of the extent to which the PD 
offered increased their instructional practice over time. Results of LGC revealed that, within SEED 
schools, teacher report of instructional-focused PD increased significantly over time (6 = 0.16, SE = 
0.02, p < 0.001). 


Instruction-Focused PD 


4.26 


2016 (Baseline) 2017 (Y1 Full 2018 (Y2 Full 2019 (Y3 Full 2020 (Y4 Full 
aave)Kevaatsvaieclateyay) Implementation) — Implementation) Tanve)kevantetarcialeyay} 


Item scale—Each item was rated on a 6-point Likert-type scale, with the responses ranging from strongly disagree (1) to strongly 


agree (6). 

Items measuring instructional-focused PD: e The PD encouraged me to reflect on aspects of my teaching. 

e __| learned new and different ideas from my PD e The PD gave me useful ideas of how to improve student 
experiences. outcomes. 

e The PD improved my knowledge of instructional e The PD updated my professional knowledge. 
strategies. e The PD was offered deepened my content knowledge. 

e Knowledge gained from the PD improved my e The PD | was offered was differentiated to meet my individual 
teaching skills. needs. 


e The PD increased my enthusiasm for teaching. 
Figure 15. Instructional-Focused PD Over Time (Teacher Survey) 


Figure 16 shows the average rating of teacher perception of the extent to which the PD offered 
supports student learning over time. Results of LGC revealed that, within SEED schools, school- 
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level mean of teacher report of student learning-focused PD increased significantly over time (6 = 
0.16, SE = 0.02, p < 0.001). When comparing the ratings between SEED and non-SEED teachers 
using Y3 data, SEED teachers gave higher ratings on the construct compared to non-SEED 
teachers (6 = 0.58, SE = 0.11, p < 0.001). 


Student Learning-focused PD 


2016 (Baseline) 2017 (Y1 Full 2018 (Y2 Full 2019 (Y3 Full 2020 (Y4 Full 
ifanvel(cYantelareciucelay} aave)Kevaatsvaieclatelay) ifaave)(cYantevaieciucelay) Implemenation) 


Item scale—Each item was rated on a 6-point Likert-type scale, with the responses ranging from strongly disagree (1) to strongly 
agree (6). 

Items measuring student-learning focused PD: 

e The PD | was offered enhanced my ability to implement instructional strategies that meet diverse student learning needs. 

e The PD | was offered provided ongoing opportunities for me to work with colleagues to refine teaching practices. 

e The PD | was offered enhanced my abilities to improve student learning. 


Figure 16. Student Learning-focused PD Over Time (Teacher Survey) 


Goal 4b. Teachers demonstrated practice change in the classroom as a result of SEED 
based on teacher report. Through the online survey, teachers answered several questions related to 
this topic. They were (1) teacher perception of SEED’s impact on teacher instruction and student 
learning; (2) teacher application of what was learned from SEED in the classroom; (3) teacher 
perception of SEED’s impact on collaboration practice; and (4) opportunities to collaborate with 
peers. Teacher evaluation ratings were also analyzed. Findings are summarized as follows. 


Teachers’ perceptions of SEED impact on teaching and learning incteased over 
time. Figure 17 shows the average rating of teachers' perceived SEED impact on teaching 
and learning over time. Only SEED participants (see the definition of SEED participants in 
Appendix A) responded to the questions. Results of LGC revealed that the school-level 
ratings from SEED teachers increased significantly over time (6 = 0.15, SE = 0.05, p = 
0.004). 
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Perceived SEED Impact on Teaching and Learning 


475 5-02 


2017 (Y1 Full 2018 (Y2 Full 2019 (Y3 Full 2020 (Y4 Full 
ifaavol(cYanteyareciulelay} Taavo)(cYanteyareciuleyay) faave)Kevantsvaieclateyay) Implemenation) 


Item scale—Each item was rated on a 6-point Likert-type scale, with the responses ranging from strongly disagree (1) to strongly 
agree (6). 
Items measuring the perceived SEED impact on teaching and learning: 

e What I learned from SEED taught me how to implement effective strategies in my classroom. 

e SEED content is directly relevant to teaching and learning in my school. 

e __|use the knowledge gained from SEED in my work with students. 

e —_| am able to put into practice the ideas presented in SEED. 

e SEED has improved student learning in my classroom. 

e SEED has provided me with knowledge and skills to support high-needs students. 


Figure 17. Perceived SEED Impact on Teaching and Learning Over Time (Teacher 
Survey) 


e Teacher application of what was learned from SEED in the classroom increased over 
time. Figure 18 shows the average rating of teachers’ applications of what was learned from 
SEED in the classroom. Only SEED participants responded to these questions. Results of 
LGC revealed that the school average ratings from SEED teachers increased significantly 
over time (8 = 0.10, SE = 0.04, p = 0.026). 


Apply What Was Learned from SEED in Classroom 


3.94 3-68 3.87 ae 


2017 (Y1 Full 2018 (Y2 Full 2019 (Y3 Full 2020 (Y4 Full 
Tanvol(cYaatevareciucela) 4 ifantol(cYanteyareciucelay) fanvel(cYanteyareciucelay) Taave)kevantsvarciaieday) 


Item scale—Each item was rated on a 6-point Likert-type scale, with the responses ranging from strongly disagree (1) to strongly 
agree (6). 
Figure 18. Apply What Has Been Learned from SEED in Classrooms*® Over Time 
(Teacher Survey) 


e Teachers’ perceptions of SEED’s impact on collaboration practice increased over 
time. Figure 19 shows the average rating of teachers’ perceived SEED impact on 
collaboration practice over time. Only SEED participants responded to these questions. 
Results of LGC revealed that the school average ratings from SEED teachers increased 
significantly over time (6 = 0.25, SE = 0.03, p < 0.001). 


6 The construct was measured by one item—On average, how often do_you apply what you have learned in SEED in your 
classrooms? 
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Perceived SEED Impact on Collaboration 


4.22 a29 


2017 (Y1 Full 2018 (Y2 Full 2019 (Y3 Full 2020 (Y4 Full 
Taste) (evantsvaieclaleyay) asve)Kedantsvaieclateyay) ifaavol(cYanteyareciuleya)) Tanve)cvantevarcialeyay} 


Item scale—Each item was rated on a 6-point Likert-type scale, with the responses ranging from strongly disagree (1) to 
strongly agree (6). 

Items measuring the perceived SEED impact on teaching and learning: 

e SEED has encouraged teachers to share what they learned with their colleagues. 

e _ Myschool provided opportunities for teachers to share information gained from SEED with colleagues. 

e Teachers in my school took advantage of opportunities to share ideas, knowledge, and skills gained from SEED. 

e SEED involved collaboration with teachers from other schools who teach at the same grade level and/or content area. 


Figure 19. Perceived SEED Impact on Collaboration Practice Over Time (Teacher 
Survey) 


e Teacher evaluation ratings increased over time. Teacher evaluation scores were 
calculated using the raw score from the teacher evaluation rubric.’ Figure 18 shows the 
average teacher evaluation scores over time. Results of LGC revealed that, within SEED 
schools, teachers’ evaluation scores, on average, increased significantly over time. (6 = 0.02, 


SE = 0.002, p < 0.001). 


Teacher Evaluation Ratings 


0-80 0-83 


2017 (Y1 Full 2018 (Y2 Full 2019 (Y3 Full 2020 (Y4 Full 
aave)Kevantsvaieclaleyay) ifanvol(cYantevareciulelay} ifaavo)(cYantelareciulelay) Tanto) (cYantevarciuleyay 


Figure 20. Teacher Evaluation Ratings Over Time 


e Teacher reports of opportunities to collaborate with job-alike peers increased. Figure 
19 shows the average teacher ratings of opportunities to collaborate with job-alike peers over 
time. Results of LGC revealed that, within SEED schools, teacher reports of the 
opportunities to collaborate with job-alike peers increased significantly over time (6 = 0.23, 
SE = 0.03, p < 0.001). 


7 The teacher evaluation score was calculated by using the total number of practices present (i.e., each practice was 
counted as 1 point) divided by the total number of practice points possible. A higher score means more desired practices 
present in the evaluation process. Data from the 2019-20 school year were not available due to the interruption of the 
COVID-19 pandemic. 
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Collaboration with job alike peers 


3.52 me) 3:57 3:63 37 


2016 (Baseline) 2017 (Y1 Full 2018 (Y2 Full 2019 (Y3 Full 2020 (Y4 Full 
aaYo)Kevaatsvaieclaleyay} ifanve)Kevanteyareciucelay) ifanve)(cYanteyareciuceyay} Implemenation) 


Item scale—Each item was rated on a 6-point Likert-type scale, with the responses ranging from strongly disagree (1) to strongly 
agree (6). 


Figure 21. Opportunity to Collaborate With Job-Alike Peers® Over Time 


Goal 4c. SEED teachers reported more favorable outcomes compared to non-SEED 
teachers in terms of consttucts related to knowledge of best practices. SEED teachers and 
non-SEED teachers did not difter in terms of demonstration of change in practice in the 
classroom. When comparing the ratings of SEED and non-SEED teachers using Y3 data, SEED 
teachers provided higher ratings on the following constructs: 


e Instructional practice focused PD (6 = 0.67, SE = 0.12, p < 0.001) 
e Student learning focused PD (6 = 0.58, SE = 0.11, p < 0.001) 


Based on teacher self-reports, SEED seemed to have a positive effect on teachers’ practice in 
classrooms as measured by the constructs of perceived impact on teaching and learning, perceived 
impact on collaboration, and application of practices learning from SEED to practice. However, 
when examining the difference on teacher evaluation ratings between SEED and non-SEED 
teachers, there was no statistically significant difference between the groups (6 = 0.01, SE = 0.01, p 
= 0.320). The opportunity to collaborate with peers also did not differ between the groups (6 = 0.11, 
SE = 0.11, p = 0.348). 

Goal 4d. HP schools, on average, had some favorable outcomes compared to LP schools in 
terms of constructs related to knowledge of best practices. HP and LP schools did not differ 
in terms of demonstration of change in practice in classtooms. When compating the school- 
level ratings between HP and LP schools, HP schools had higher ratings on the following 
constructs: 


e Instructional practice focused PD (6 = 0.31, SE = 0.09, p = 0.003) 
e Student learning focused PD (6 = 0.31, SE = 0.10, p = 0.005) 


HP and LP schools did not differ in terms of teacher ratings (@ = 0.003, SE = 0.03, p = 0.917) or the 
opportunities to collaborate with peers (6 = 0.20, SE = 0.20, p = 0.343). 


Due to the nature of the correlational design, causal inference cannot be made. Yet, findings from 
Goals 4c and 4d are encouraging as they suggest that SEED participation at teacher and school 
levels were mostly associated with teachers’ knowledge of best practices as well as perceived impact 
on practices. 


8 This construct was measured by one item. 
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Goal 5. Improve Student Achievement 


Goal 5 of SEED is to increase student achievement. Five questions were addressed: To what extent do 
SEED schools and comparison schools differ in student achievement outcomes after one year, two years, and three 
years of full implementation (Goal 5a)? To what extent do students from SEED schools demonstrate an increase in 
student achievement outcomes over time (Goal 5b)? To what extent do high-needs students from SEED schools and 
comparison schools differ in student achievement outcomes after one year, two years, and three years of full 
implementation (Goal 5c)? To what extent do high-needs students from SEED schools demonstrate an increase in 
student achievement outcomes over time (Goal 5d)? Do schools with higher levels of teacher participation have higher 
student achievement (Goal 5e)? 


Publicly available data between 2015-16 and 2018-19 from the Colorado Department of Education 
website were used in the analysis. To answer Goals 5a and 5c, regression analyses were conducted to 
examine the difference between SEED schools and their matched non-SEED” schools at the end of 
Y1, Y2, and Y3, respectively. To answer Goals 5b and 5d, conditional LGC models were conducted 
to examine the rate of change over time while controlling for baseline covariates. These questions 
were addressed using QED design with six study samples (see Evaluation Design section under 
Appendix A for more information about the study samples): 


e Study sample 1: All students with ELA outcome 

e Study sample 2: Minority students with ELA outcome 
e Study sample 3: FRL students with ELA outcome 

e Study sample 4: All students with Math outcome 

e Study sample 5: Minority students with Math outcome 
e Study sample 6: FRL students with Math outcome 


To answer Goal 5e, regression analyses were conducted to examine the differences in achievement 
outcomes between HP and LP schools. Findings are summarized in the following sections. 


Goal 5a. SEED had a marginally significant effect on student ELA scores by the end of Year 
3. A series of regression analyses was conducted to examine the impact of SEED on students’ 
reading and math outcomes by the end of Y1, Y2, and Y3, respectively. The detailed statistical 
outputs are provided in Appendix C. One positive finding was found: 


e SEED had a marginally significant effect on student ELA outcome by the end of Y3 (8 
= 0.10, SE = 0.05, p = 0.052). The effect size was 0.10. 


Other null findings are summarized as follows: 


e SEED and non-SEED schools did not differ on student ELA outcome by the end of Y1 
(6 = 0.04, SE = 0.04, p = 0.263). The effect size was 0.04. 


e SEED and non-SEED schools did not differ on student ELA outcomes by the end of 
Y2 (6 = 0.06, SE = 0.05, p = 0.197). The effect size was 0.06. 


e SEED and non-SEED schools did not differ on student math outcome by the end of 
Y1 (6 = 0.04, SE = 0.04, p = 0.337). The effect size was 0.04. 

e SEED and non-SEED schools did not differ on student math outcomes by the end of 
Y2 (6 = 0.02, SE = 0.04, p = 0.634). The effect size was 0.02. 


° The matching methods and processes are reported in detail in the Study Sample section in Appendix A. 
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e SEED and non-SEED schools did not differ on student math outcomes by the end of 
Y3 (6 = 0.03, SE = 0.05, p = 0.634). The effect size was 0.03. 


Goal 5b. SEED schools maintained the same level of ELA scores over time, while non- 
SEED schools showed a statistically significant decline on ELA scores over time. Both 
SEED and non-SEED schools showed a minor decline (not statistically significant) in math 
outcomes over time. Figures 20 and 21 show the unadjusted SEED schools’ average ELA and 


math scores, respectively, over time. 


SEED Schools = —<=Comparison Schools 


2016 (Baseline) 2017 (Y1 Full 2018 (Y2 Full 2019 (Y3 Full 
Tanvo)(cYantevaneciuceyap Mmmm (nal) (elaatsvalecidio)a) Mumm loa) e)(cleatlale-iaveye)) 


Figure 22. SEED and Non-SEED Schools’ Unadjusted Mean ELA Scores Over Time 


AS} =f =] DJNSYo1 a oo) [I (0) 00) oF] 8110) glmso1 aol) [5 


2016 (Baseline) 2017 (Y1 Full 2018 (Y2 Full 2019 (Y3 Full 
Tanvo)(cYantevaneciuleyap Mmmm (nae) (elpelstalecid(o)a) Mumm lea} e)(cteatelale-vavele)) 


Figure 23. SEED and Non-SEED Schools’ Unadjusted Mean Math Scores Over Time 


A series of multilevel models was conducted to examine if the rate of change in achievement, if any, 
differed by group." Key findings are summarized below: 


10 All covariates used in the matching were also included in the analyses. 
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e Non-SEED schools demonstrated a significant rate of decrease in ELA outcomes over time 
(6 = -0.02, SE = 0.01, p = 0.004), and the rate of change was marginally different between 
SEED and non-SEED schools (6 = 0.03, SE = 0.02, p = 0.078). In other words, while non- 
SEED schools seemed to decline in ELA outcomes over time, SEED schools seemed to 
maintain a consistent level of achievement over time. 


e Non-SEED schools demonstrated a marginal rate of decrease in math outcomes over time 
(6 = -0.01, SE = 0.01, p = 0.071), and the rate of change was not different between SEED 
and non-SEED schools (6 = 0.00, SE = 0.02, p = 0.964). In other words, both SEED and 
non-SEED schools seemed to have a minor decline in math outcomes over time. 


Goal 5c. SEED increased minority students’ ELA outcome by the end of Y1. SEED also had 
marginally significant effects on minority students’ math outcome by the end of Y1, and 
FRL students’ ELA and math outcomes by the end of Y2. A series of regression analyses was 
conducted to examine the impact of SEED on minority and FRL students’ reading and math 
outcomes by the end of Y1, Y2, and Y3, respectively. The detailed statistical outputs are provided in 
Appendix C. Four positive findings were found: 


SEED had a statistically significant effect on minority students’ ELA outcomes by the 
end of Y1 (6 = 0.20, SE = 0.06, p = 0.002). The effect size was 0.20. 

SEED had a marginally significant effect on FRL students’ ELA outcomes by the end of 
Y2 (6 = 0.08, SE = 0.04, p = 0.054). The effect size was 0.08. 

SEED had a marginally significant effect on minority students’ math outcomes by the 
end of Y1 (6 = 0.09, SE = 0.05, p = 0.072). The effect size was 0.09. 

SEED had a marginally significant effect on FRL students’ math outcomes by the end of 
Y2 (6 = 0.09, SE = 0.05, p = 0.099). The effect size was 0.09. 


Other null findings are summarized as follows: 


Minority students within SEED schools and minority students within non-SEED 
schools did not significantly differ on ELA outcome by the end of Y2 (6 = -0.01, SE = 
0.07, p = 0.933). The effect size was 0.01. 

Minority students within SEED schools and minority students within non-SEED 
schools did not significantly differ on ELA outcome by the end of Y3 (6 = 0.04, SE = 
0.07, p = 0.581). The effect size was 0.04. 

PRL students within SEED schools and FRL students within non-SEED schools did 
not significantly differ on ELA outcome by the end of Y1 (@ = 0.07, SE = 0.05, p = 
0.184). The effect size was 0.07. 

PRL students within SEED schools and FRL students within non-SEED schools did 
not significantly differ on ELA outcome by the end of Y3 ( = 0.03, SE = 0.05, p = 
0.564). The effect size was 0.03. 

Minority students within SEED schools and minority students within non-SEED 
schools did not significantly differ on math outcome by the end of Y2 (6 = 0.03, SE = 
0.06, p = 0.656). The effect size was 0.03. 

Minority students within SEED schools and minority students within non-SEED 
schools did not significantly differ on math outcome by the end of Y3 (6 = 0.11, SE = 
0.07, p = 0.152). The effect size was 0.11. 
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e FRL students within SEED schools and FRL students within non-SEED schools did 
not significantly differ on ELA outcome by the end of Y1 ( = 0.06, SE = 0.05, p = 
0.257). The effect size was 0.006. 

e FRL students within SEED schools and FRL students within non-SEED schools did 
not differ on ELA outcome by the end of Y3 (6 = 0.03, SE = 0.05, p = 0.535). The 
effect size was 0.03. 


Goal 5d. Both minority and FRL students within SEED and non-SEED schools maintained 
the same levels of ELA achievement over time. FRL students within SEED and non-SEED 
schools also maintained the same level of math achievement over time. Minority students 
within SEED and non-SEED schools both showed significant levels of decline in math 
achievement over time. Figures 22 and 23 show the unadjusted mean ELA scores over time 
between minority students within SEED and non-SEED schools, and FRL students within SEED 
and non-SEED schools, respectively, over time. 


Minority Student in SEED Schools —— Minority Students in Non-SEED Schools 


2016 (Baseline) 2017 (Y1 Full 2018 (Y2 Full 2019 (Y3 Full 
aave)(edaatsvaieclateyay) Tanve)cvanteyaieciuleyay) ifaavol(cYantevareciulela)) 


Figure 24. Minority Students in SEED and Non-SEED Schools’ Unadjusted Mean 
ELA Scores Over Time 


FRL Students in SEED Schools ——FRL Students in Non-SEED Schools 


2016 (Baseline) 2017 (Y1 Full 2018 (Y2 Full 2019 (Y3 Full 
anve)Kevantsvaieclateyay) ifaavo)Kevanteyareciulelay} fanve)(cYanteyareclalelay) 


Figure 25. FRL Students in SEED and Non-SEED Schools’ Unadjusted Mean ELA 
Scores Over Time 
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Figures 24 and 25 show the unadjusted mean math scores over time between minority students 


within SEED and non-SEED schools, and FRL students within SEED and non-SEED schools, 
respectively, over time. 


Minority Student in SEED Schools =——— Minority Students in Non-SEED Schools 


-0.34 


2016 (Baseline) 2017 (Y1 Full 2018 (Y2 Full 2019 (Y3 Full 
aave)(edaatsvaieclateyay) Taave)(cvanteyareciuleya)) ifaave)(cYanteyareciulelay) 


Figure 26. Minority Students in SEED and Non-SEED Schools’ Unadjusted Mean 
Math Scores Over Time 
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Figure 27. FRL Students in SEED and Non-SEED Schools’ Unadjusted Mean Math 
Scores Over Time 


A series of multilevel models was conducted to examine if the rate of change in achievement, if any, 
differed by groups.'' Key findings are summarized below: 


e Minority students within non-SEED schools did not demonstrate significant change in ELA 


scores over time (8 = -0.00, SE = 0.03, p = 0.911), and the rate of change was not 
significantly different between minority students within SEED and those within non-SEED 
schools (6 = -0.01, SE = 0.04, p = 0.763). In other words, minority students within both 


SEED and non-SEED schools, on average, maintained the same level of ELA achievement 
overt time. 


1! All covariates used in the matching were also included in the analyses. 


37 


e FRL students within non-SEED schools did not demonstrate significant change in ELA 
scores over time (8 = -0.01, SE = 0.01, p = 0.673), and the rate of change was not different 
between FRI students within SEED and those within non-SEED schools (6 = 0.02, SE = 
0.02, p = 0.430). In other words, FRL students within both SEED and non-SEED schools, 
on average, maintained the same level of ELA achievement over time. 

e Minority students within non-SEED schools demonstrated a significant rate of decrease in 
math outcome over time (6 = -0.05, SE = 0.02, p = 0.037), and the rate of change was not 
significantly different between minority students in SEED schools and those in non-SEED 
schools (8 = 0.02, SE = 0.03, p = 0.492). In other words, minority students in SEED 
schools and non-SEED schools both demonstrated significant decline in ELA outcomes 
over time. 

e FRL students within non-SEED schools did not demonstrate significant change in math 
outcome over time (6 = 0.00, SE = 0.01, p = 0.793), and the rate of change was not 
significantly different between FRL students within SEED and those within non-SEED 
schools (6 = 0.02, SE = 0.02, p = 0.431). In other words, FRL students within both SEED 
and non-SEED schools, on average, maintained the same level of math achievement over 
time. 


Goal 5d. HP and LP schools did not differ on student achievement scores by the end of Y3. 
Regression analyses were conducted to examine the differences in achievement outcomes, including 
ELA and math, between HP and LP schools using Y3 data. Results showed that HP and LP schools 
did not differ on ELA (6 = 0.13, SE = 0.09, p = 0.219) or math (6 = 0.04, SE = 0.17, p = 0.839) by 
the end of Y3. 


Goal 6. Improve Student Engagement 


Goal 6 of SEED was to improve student engagement. Two questions were examined: Does student 
engagement increase among participating schools (Goal 6a)? Do schools with higher levels of teacher participation have 
higher levels of student engagement (Goal 6b)? Data collected from the student engagement survey between 
the 2016-17 and the 2018-19 were included in the analyses.’ To answer Goal 6a, a series of LGC 
was conducted to examine the rate of change on each of the measured constructs over time. School- 
level means were used in the analyses. To answer Goal 6b, linear regression analysis was conducted 
to examine the difference between HP and LP schools on the outcomes of interest. Findings are 
summarized in the following sections. 


Goal 6a. Levels of student engagement largely remained unchanged over time with some 
subscales showing decteases over time. Starting with the baseline of full implementation (2016), 
students in grades 3-5 answered a series of questions related to their sense of school engagement. 
These questions measured four constructs: 


e Teacher-Supported Engagement: Individual Student Learning (4 items) 
e Teacher-Supported Engagement: Content Learning (5 items) 

e Positive Attitude Toward School (4 items) 

e Teacher Support (4 items) 


Figure 24 is a visual representation of student engagement ratings across the four constructs over 
time. Overall, these ratings remained high throughout the implementation period. Results of LGC 
revealed that, within SEED schools, student ratings on all constructs largely remained unchanged 
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over time with one exception—student reports of positive attitude toward school decreased 
significantly over time. 


e Teacher-Supported Engagement: Individual Student Learning (6 = -0.01, SE = 0.02, 
p = 0.452) 

e Teacher-Supported Engagement: Content Learning (6 = -0.01, SE = 0.02, p = 0.572) 

e Positive Attitude Toward School (6 = -0.02, SE = 0.01, p = 0.019) 

e Teacher Support (6 = -0.003, SE = 0.01, p = 0.494) 


2016 (Baseline) 012017 (Y1 Full Implementation) 


012018 (Y2 Full Implementation) 0 2019 (Y3 Full Implementation) 


Teacher Supported Engagement: Individual Student 
Learning 


Teacher Supported Engagement: Content Learning 


Positive Attitude toward School 


Teacher Support 


Note: Teacher support and positive attitude toward school were rated Yes (1) or No (0), and the two teacher-supported 
engagement constructs were rated on a 5-point scale, ranging from strongly disagree (1) to strongly agree (5). 


Figure 28. Student Engagement Ratings Over Time: Grades 3-5 
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Student survey report of student engagement: Grades 6-10. Since the baseline of the full 
implementation (2016), students in grades 6-10 answered a series of questions related to their sense 
of school engagement. These questions measured five constructs: 


School Engagement in Math Learning (4 items) 

School Engagement in English/Language Arts Learning (4 items) 
Teacher-Supported Engagement: Individual Student Learning (4 items) 
Teacher-Supported Engagement: Content Learning (5 items) 
Engagement in Academic Success (5 items) 


Figure 25 is a visual representation of student engagement ratings across five constructs over time. 
Results of LGC revealed that, within SEED schools, student ratings on three constructs decreased 
significantly over time. They were: school engagement in math learning, teacher-supported 
engagement: individual student learning, and engagement in academic success. The results follow. 


School Engagement in Math Learning (6 = -0.08, SE = 0.02, p < 0.001) 

School engagement in English/language arts Learning (6 =-0.07, SE = 0.03, p = 
0.059) 

Teacher-Supported Engagement: Individual Student Learning (6 = -0.07, SE = 0.03, 
p= 0.012) 

Teacher-Supported Engagement: Content Learning (6 = -0.04, SE = 0.03, p = 0.203) 
Engagement in Academic Success (8 = -0.09, SE = 0.03, p = 0.006) 
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2016 (Baseline) 12017 (Y1 Full Implementation) 


012018 (Y2 Full Implementation) 0 2019 (Y3 Full Implementation) 


School Engagement in Math Learning 


School Engagement in English/Language Arts 
Learning 


Teacher Supported Engagement: Individual 
Student Learning 


Teacher Supported Engagement: Content 
Learning 


Engagement in Academic Success 


Notes: School engagement in math leaning and school engagement in English/language arts learning constructs include four 
items; one is measured on a 6-point scale, and three are measured on a 5-point scale. Teacher-supported engagement 
constructs were measured on a 5-point scale. Engagement in academic success was measured by four items on a 7-point scale. 


Figure 29. Student Engagement Ratings Over Time: Grades 6-10 
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Goal 6b. HP and LP schools did not differ in terms of levels of student engagement actoss 
most of the constructs with one positive exception—HP schools had higher ratings from 6" 
to 10" graders regarding their perception of teacher-supported engagement: Individual 
student learning. When comparing the school-level ratings between HP and LP schools, they were 
similar on most of the student engagement subscales with one exception. That is, students from 6" 
to 10 grades within HP schools reported a higher level of teacher-supported engagement on 
individual student learning compared to LP schools. Results from the analyses are summarized as 
follows. 


e Students from grades 3 to 5 (SES: 3-5") 
© Teacher-Supported Engagement: Individual Student Learning (6 = 0.25, SE = 


0.08, p = 0.096) 
o Teacher-Supported Engagement: Content Learning (6 = 0.23, SE = 0.12, p = 
0.197) 


© Positive Attitude Toward School (6 = 0.12, SE = 0.07, p = 0.251) 
Oo Teacher Support (6 = 0.05, SE = 0.06, p = 0.508) 


e Students from grades 6 to 10 (SES: 6-10") 
© School Engagement in Math Learning (6 = 0.18, SE = 0.21, p = 0.461) 
o School Engagement in English/Language Arts Learning (6 =0.14, SE = 0.12, p 


= 0.302) 

oO Teacher-Supported Engagement: Individual Student Learning (6 = 0.25, SE = 
0.03, p = 0.003) 

o Teacher-Supported Engagement: Content Learning (6 =0.26, SE = 0.17, p = 
0.213) 


© Engagement in Academic Success (6 =0.08, SE = 0.20, p = 0.698) 
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Conclusions and Recommendations 


SEED was designed to achieve six goals. Through one year of planning and pilot testing (2015-16) 
and four years of full implementation (2016-17 to 2019-20), SEED achieved all six of its goals. 
Quotes from the teacher participants below highlight the value of SEED. 


This is a GREAT way to re-energize and remind yourself of best practices. | found the resources interesting and thought 
provoking. | implemented elements in my classes each week of the SEED course. | found that the class helped with daily 
lesson planning for lessons that hopefully increased student learning and engagement. 


This course helped me evolve as a teacher in a world of evolving learners. | feel better prepared to teach my students 
through a collaborative approach and with collaborative strategies. | have more confidence in my teaching and 
modeling of collaboration, and | am thrilled at the student outcomes and learning from their collaboration. 


The changes that I've made as a result of this course have been instrumental to my teaching practice and to my 
students’ increased learning experiences and understanding. These changes will continue throughout my teaching 
career. This course provided me the guidance and the structure to feel safe to take risks and make these changes in my 
classroom. 


| am a total SEED nerd and love these classes! They all build upon one another and help me improve my practice a little 
more each semester. | am especially excited about how | am pulling in more of my outliers and encouraging and inviting 
more participation from all students. The coaching sessions are especially helpful to keep me focused and problem 
solve the classroom implementation between modules. Thank you for providing such positive and relevant professional 
development for teachers! 


| absolutely treasure the pacing of the SEED course. We read insightful resources that cause us to think. We then 
integrate assignments in the class over the course of a semester. This is So powerful because we are integrating 
strategies with our real-life kiddos. | feel like every single SEED class | take has great impact on my students because 
it gives me a focal point and goal. Additionally, the reflection piece is so powerful because | take my new tools with me. 


The TLC courses are so helpful and integrate right into your classroom instruction. It is easy to learn new things 
while not feeling overwhelmed by your own planning and the classwork. 


The workload required for the TLC courses is manageable while teaching during the school year. The changes that I’ve 
made while taking this course have been invaluable to my students’ understanding and to my teaching practice. | 
was able to implement changes that I’ve been thinking about making for a while but needed the support and guidance. 
The TLC provided me with that through reading resources, peer support, and coaching conversations. 


| loved this course. [ICs] are always super supportive and provide great insight and direction for how to push my thinking. 
| love the classroom implementation component of the TLC and all the SEED courses so that | am pushed to take 
what | learn and put it into practice. | can’t wait to be back in my classroom guiding my class and myself to be better 
learners. 


My TLC experience was meaningful and fit into my schedule as a teacher. It was wonderful because it directly relates 
and can be incorporated into your current classroom. It is a great use of time for getting more PD. 


| would say that it is valuable. You learn authentic educational material, and you can immediately implement what you 
have learned in the classroom. Also, the way it is tied to the state evaluation rubric is very helpful when designing your 
individual goals for the year. | would definitely recommend the course. 


—SEED participants 
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Implementation data also showed that SEED was successfully implemented and continually 
improved in 21 schools throughout the project period. Seven out of eight fidelity indicators (88%) 
achieved the desired level of implementation. Additionally, SEED significantly increased the short- 
term outcomes as depicted in the logic model (see Figure ES1), and ultimately has provided some 
positive and promising pieces of evidence supporting its desired long-term outcome—student 
achievement. Specifically, SEED significantly increased student achievement among minority 
learners after one year of implementation. Additionally, SEED had some marginal effects across 
various groups and outcomes, including all students’ ELA outcome in Y3; FRL students’ ELA and 
math outcomes in Y2; and minority students’ math outcome in Y1. When examining the trajectories 
of student achievement between SEED and non-SEED schools, results showed that while non- 
SEED schools showed a statistically significant decline on ELA scores over time, SEED schools 
maintained the same level of ELA scores over time. 


Considering these effects within the context that the desired level of teacher participation was not 
achieved until Y3, the observed effects were encouraging. According to SEED participation data, 
the school-level SEED participation rate (percentage of teachers participating in SEED) was 16% 
(ranging from 0% to 63%), 20% (ranging from 6% to 48%), 33% (ranging from 7% to 69%), and 
40% (ranging from 10% to 100%) in Y1, Y2, Y3, and Y4, respectively. Figure 26 shows the school- 
level participation rate over time; participation levels within schools fluctuated over time, and there 
were noticeable variations in terms of participation rates across schools. One may imagine the 
potential impact of SEED if the participation level was higher and more consistent throughout the 
participating schools. 


KY ol YoYo) fy K=\(-] ms) ol ol DN e-Tad (all oy- lala) 


School 3 =———=School 4 ——<=School5 =——<=School 6 School 7 Yel afoe) its} Sol aoe) Io) 
Yel aTole) i) School 11 = School 12 School 13 School 14 Yel aoe) fat bo) Yel aloe) Into) 
School 17 School 18 School 19 Yel solo) i720) School 21 Wiel ee} xc) 


Figure 30. School-Level SEED Participation Over Time 


This evaluation provided some positive and promising pieces of evidence to support SEED’s theory 
of action. In Figure 27, the solid blue arrows illustrate pathways that were supported by the 
evaluation. More research is needed to establish the links between the short-term (Goals 2—4 and 6) 
and long-term outcomes (Goal 5). The pathways illustrated by orange dashed arrows were not tested 
in this evaluation. Future studies examining the mediation pathways presented in Figure 31 will 
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provide valuable information regarding the understanding of the mechanisms through which 
programs like SEED work to support student achievement. 


Principal Engagement 


Access to and use of best 
practice and up-to date 
Successful content knowledge 
SEED 


Increased 
Implementation 


Student 


Increased knowledge and we Achievement 
Implement SEED practice 


Increased student 
engagement 


Figure 31. The Pathways Through Which SEED Exerts Its Influence on Student 
Achievement 
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Appendices 


Appendix A-1 


Appendix A. SEED Evaluation: Technical Report 


The System for Educator Effectiveness (SEED) was a six-year project (2015-2020) funded by the 
U.S. Department of Education, Office of Improvement and Innovation, Investing in Innovation (13) 
program. SEED was led by the Northwest Colorado Board of Cooperative Educational Services 
(NW BOCES). McREL International (McREL) was the third-party evaluation partner for the 
project. Figure Al shows an overview of the SEED project phases, implementation components, 
and the overarching goals. 


Phase | (spring 2015): Program development 
SEED Phases Phase II (2015-16 school year): Pilot testing and baseline data collection 
Phase Ill (2016-17 to 2019-20 school years): Full implementation 


Create evidence-based PD driven by teacher needs and interests 
Deliver PD through a dynamic PD platform that enables online PD and 
blended learning TLC 

Develop a repository of online, up-to-date, evidence-based PD resources 
aligned with Educator Effectiveness standards 

Engage principals and teachers in teacher PD 


Components 


Goal 1: Successfully implement and continually improve the SEED project 
in up to 30 NW BOCES schools during a four-year period 

Goal 2: Improve principal engagement in teacher professional growth and 
support 

Goal 3: Increase rural teachers’ access to and use of current best 
practices and up-to-date content knowledge 

Goal 4: Support teachers in successfully implementing SEED practices 
and content 

Goal 5: Improve student achievement 

Goal 6: Improve student engagement 


SEED Goals 


Figure Al. SEED Project Phase, Components, and Goals 


Figure A2 shows the links between SEED components (strategies), outputs, short-term outcomes, 
and long-term outcomes that are aligned with the six specified project goals. SEED strategies 
presented in the logic model were guided by the research on evidence-based PD for rural educators 
(Darling-Hammond et al., 2017; Gaumer Erickson et al., 2012; Howley & Howley, 2005; Koonce et 
al., 2019; Vescio et al., 2008). It was hypothesized that, if all elements of SEED were implemented as 
planned (.e., high fidelity; Goal 1), SEED would increase principal and teacher engagement in 
teacher PD (Goal 2); increase teacher access to evidence-based PD (Goal 3); and increase teacher 
implementation of knowledge and skills learned from PD in the classroom (Goal 4). These changes 
in classroom practice would result in improved student engagement (Goal 5) and then student 
achievement (Goal 6). 
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Short-term Long-term 


SEED Strategies Outputs 


Outcomes Outcomes 
SEED Creation e SEED e Increased principal and e Improved 
e Create evidence-based PD strategies were teacher engagement in student 
interests i els Ine Increased teacher access to (Goal 5) 
e Develop a repository of online, i evidence-based PD (Goal 3) 


schools (Goal 


1) 


up-to-date, evidence-based PD 
resources aligned with Educator 


Increased teacher application 
of knowledge and skills 


Effectiveness standards learned from PD in classroom 
SEED Delivery (Goal 4) 
e Deliver PD through a dynamic e Improved student 
PD platform that enables online engagement (Goal 6) 
PD and blended learning TLC 
SEED Participation 
e Engage principals and teachers 
in teacher PD 


Figure A2. SEED Logic Model 


The purpose of this appendix is to provide detailed information regarding the evaluation plan, 
including evaluation questions, evaluation design, study sample, data collection methods, and data 
analysis plan. Each element is discussed in detail in the following sections. 


Evaluation Questions 


Evaluation questions were guided by the SEED logic model and were designed to address each of 
the specified project goals. Table Al shows the alignment of the evaluation questions, evaluation 
designs, and data collection methods by project goal. Abbreviations used in Table A1 are defined as 
follows. 


Evaluation Design 


P = Process evaluation 

O = Outcome evaluation 

QED = Quasi-experimental design 
C = Correlational study 


DEV re Mexe) | (sVouetoya Mm /cieaveye bs 


PR = Project records TI = Teacher interview 

ICI = Innovation Coaches (IC) interview TS = Teacher survey 

PI = Principal interview SES = Student engagement survey 
PS = Principal survey ED = Extant data from the CDE 
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Questions specified under Goal 1 were designed to understand the extent to which the elements of 
SEED were implemented as intended with planned rigor; what the challenges and successes were; 
and what revisions or modifications were made during the implementation period and for what 
reasons. Questions specified under Goals 2, 3, 4, and 6 aimed to understand how and the extent to 
which the project improved the short-term outcomes as specified in the logic model, including (1) 
improved principal engagement in teacher professional growth and support (Goal 2); (2) increased 
rural teachers’ access to and use of current best practices and up-to-date content knowledge (Goal 
3); (3) increased implementation of SEED practices and content (Goal 4); and (4) increased student 
engagement (Goal 6). Questions for Goal 5 were designed to understand the extent to which the 
project achieved the long-term outcome of improved student achievement, SEED’s ultimate goal. 


Table Al. Evaluation Questions, Evaluation Design, and Data Collection Methods 
by Project Goals 


Evaluation 


Design 2 


Evaluation Questions 0 exo) | [=e t(eya) 
>) I euateye 


C QED 
Goal 1. Successfully implement and continually improve SEED in up to 30 NW BOCES schools during a 
four-year period (timeliness and continuous improvement) 
a. Are project activities occurring as planned? X PR, ICI, Pl, 
b. Is the project maintaining its planned rigor? X PS, TI, TS, 
c. What barriers and unanticipated outcomes were encountered, and what X TS 
X 


revisions were made during implementation? 
d. What were key project successes during implementation? 
Goal 2. Improve principal engagement in teacher professional growth and s 


a. Are principals more engaged in teacher professional growth? X Pl, PS, TI, TS 
b. Are principals able to provide more and better support to teachers through Xx 
SEED? 
c. Do SEED teachers, in comparison with non-SEED teachers, report a X TS 
higher level of principal engagement in teacher professional growth?* 
d. What is the relationship between school-level SEED participation rates and Xx PS, TS 


principal and teacher reports of principal engagement in teacher PD?* 
Goal 3. Increase rural teachers’ access to and use of current best practices and up-to-date content 


knowledge 
a. Are teachers utilizing SEED? Which activities are teachers using? Does X PR, TS 
use increase over time? 


. ls PD reflective of best practices and up-to-date content knowledge? PI, PS, TI, TS 

c. Do teachers have time and resources to access PD? 

d. Is PD offered to teachers driven by data and does it meet the needs of 
individual teachers? 

e. Do SEED teachers, in comparison with non-SEED teachers, give higher PS, TS 
ratings regarding the extent to which (1) PD is reflective of best practice 
and up-to date content knowledge; (2) PD offered is data-driven and X 
aligned with individual needs; and (3) teachers have time and resources to 
access PD?* 

f. What is the relationship between school-level SEED participation rates and PS, TS 
teachers’ ratings on the extent to which (1) PD is reflective of best practice X 
and up-to date content knowledge; (2) PD offered is data-driven and 


><| >< 
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Evaluation 
Design 


Data 
Evaluation Questions 0 (0X0) | {=Yeu (0) 9) 
C QED I Eeuateye 


aligned with individual needs; and (3) teachers have time and resources to 
access PD?* 

Goal 4. Support teachers in successfully implementing SEED practices and 

a. Does teachers’ knowledge of best practices increase? X TS 

b. Do teachers demonstrate practice change in the classroom? X Tl, TS, ED 

c. Do SEED teachers, in comparison with non-SEED teachers, (1) give higher TS, ED 
ratings on their knowledge of best practices, (2) have more opportunity to 
collaborate with job-alike peers, and (3) have higher teacher evaluation 
scores?* 

d. What is the relationship between school-level SEED participation rates and TS, ED 
(1) teachers' ratings on their knowledge of best practices, (2) teachers’ X 
opportunity to collaborate with job-alike peers, and (3) teacher evaluation 
scores?* 

a. To what extent do SEED schools and comparison schools differ in student ED 
achievement outcomes after one year, two years, and three years of full X 
implementation? 

b. To what extent do students from SEED schools demonstrate an increase in x ED 
student achievement outcomes over time?* 

c. To what extent do high-needs students from SEED schools and ED 
comparison schools differ in student achievement outcomes after one year, X 
two years, and three years of full implementation? 

d. To what extent do high-needs students from SEED schools demonstrate an ED 
increase in student achievement outcomes over time?* 

e. Do schools with higher levels of teacher participation have higher overall Xx 
student achievement?* 

a. Does student engagement increase among participating schools? X SES 

b. Do schools with higher levels of teacher participation have higher levels of x PR, TS, SES 
student engagement?* 

Note. In 2017-18 (grant Year 3), the data collection plan was strategically revised to reduce the redundancy of information 

collected over time. Specifically, principal, teacher and IC interviews were removed from the evaluation data collection plan. 

The teacher survey was revised to include questions related to teacher perceptions of principal engagement in teacher PD 

that were originally collected through teacher interview. In this table, additional questions added in the 2017-18 reporting 

period are marked with an asterisk. 


PR, TS, ED 


Evaluation Design 


The SEED evaluation included both a process evaluation and an outcome evaluation to address 
the proposed evaluation questions. Each approach is discussed in the following sections. 


Process Evaluation Design 
The process evaluation focused on documenting and describing the services, activities, and 


procedures of SEED to have a better understanding about the implementation fidelity and 
challenges encountered (Questions 1a, 1b, 1c and 1d), as well as teacher utilization of SEED 
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(questions 3d and 3e). Additionally, as a part of the process evaluation, data on the project’s progress 
in achieving short-term outcomes on an annual basis were examined and provided to the project 
team to inform decisions regarding program improvement and refinement, as needed, to ensure the 
project goals were achieved. As shown in Table A1, various data sources were analyzed and 
triangulated to provide an in-depth and comprehensive understanding of program implementation. 
Additionally, an implementation fidelity matrix was created to understand and monitor the extent to 
which SEED strategies were implemented as planned and with the intended rigor. The fidelity 
matrix is presented in the Data Collection Methods section under Fidelity Assessment. 


Outcome Evaluation Design 


The outcome evaluation was conducted to understand the extent to which SEED had a positive 
effect on principal engagement in teacher PD (Goal 2), teacher access to and use of best practices 
and up-to-date content knowledge (Goal 3), teacher implementation of SEED practice and content 
in the classroom (Goal 4), student achievement (Goal 5), and student engagement (Goal 6). As 
shown in Table A2, different study designs were employed to address different sets of questions. 


Table A2. Evaluation Questions and Research Design by Project Goal 


Evaluation Questions 


Research Design 


Goal 1. Successfully implement and continually improve SEED in up to 30 NW BOCES schools during a 


four-year period (timeliness and continuous improvement) 
a. Are project activities occurring as planned? 


b. Is the project maintaining its planned rigor? 


c. What barriers and unanticipated outcomes were encountered, and 
what revisions were made during implementation? 


d. What were key project successes during implementation? 


a. Are principals more engaged in teacher professional growth? 


b. Are principals able to provide more and better support to teachers 
through SEED? 


Process evaluation 


Goal 2. Improve principal engagement in teacher professional growth and support 


Correlational design—single group 
with repeated measures 


c. Do SEED teachers, in comparison with non-SEED teachers, report 
a higher level of principal engagement in teacher professional 
growth?* 


Correlational design—SEED and non- 
SEED teacher comparisons 


d. What is the relationship between school-level SEED participation 
rates and principal and teacher reports of principal engagement in 
teacher PD?* 


knowledge 
a. Are teachers utilizing SEED? Which activities are teachers using? 
Does use increase over time? 


Correlational design—ow- and high- 
participation school comparisons 


Goal 3. Increase rural teachers’ access to and use of current best practices and up-to-date content 


Process evaluation 


b. Is PD reflective of best practices and up-to-date content 
knowledge? 


c. Do teachers have time and resources to access PD? 


d. Is PD offered to teachers driven by data and does it meet the needs 
of individual teachers? 


Correlational design—single group 
with repeated measures 


e. Do SEED teachers, in comparison with non-SEED teachers, give 
higher ratings regarding the extent to which (1) PD is reflective of 
best practice and up-to date content knowledge; (2) PD offered is 


Correlational design—SEED and non- 
SEED teacher comparisons 
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Evaluation Questions Research Design 


data-driven and aligned with individual needs; and (3) teachers 
have time and resources to access PD?* 


What is the relationship between school-level SEED participation 
rates and teachers’ ratings on the extent to which (1) PD is 
reflective of best practice and up-to date content knowledge; (2) PD 
offered is data-driven and aligned with individual needs; and (3) 
teachers have time and resources to access PD?* 


Correlational design—low- and high- 
participation school comparisons 


Goal 4. Support teachers in successfully implementing SEED practices and content 


rates and (1) teachers’ ratings on their knowledge of best practices, 
(2) teachers’ opportunity to collaborate with job-alike peers, and (3) 
teacher evaluation scores?* 


To what extent do SEED schools and comparison schools differ in 
student achievement outcomes after one year, two years, and three 
years of full implementation? 


To what extent do students from SEED schools demonstrate an 
increase in student achievement outcomes over time?* 


To what extent do high-needs students from SEED schools and 
comparison schools differ in student achievement outcomes after 
one year, two years, and three years of full implementation? 


To what extent do high-needs students from SEED schools 
demonstrate an increase in student achievement outcomes over 
time?* 


a. Do teachers’ knowledge of best practices increase? Correlational design—single group 

b. Do teachers demonstrate practice change in the classroom? with repeated measures 

c. Do SEED teachers, in comparison with non-SEED teachers, (1) Correlational design—SEED and non- 
give higher ratings on their knowledge of best practices, (2) have SEED teacher comparisons 
more opportunity to collaborate with job-alike peers, and (3) have 
higher teacher evaluation scores?” 

d. Whatis the relationship between school-level SEED participation Correlational design—low- and high- 


participation school comparisons 


Goal 5. Improve student achievement 


a. 


QED with a matched comparison 
group 


e. 


Do schools with higher levels of teacher participation have higher 
overall student achievement?* 


Goal 6. Improve student engagement 


Correlational design—low- and high- 
participation school comparisons 


levels of student engagement?* 


a. Does student engagement increase among participating schools? Correlational design—single group 
with repeated measures 
b. Do schools with higher levels of teacher participation have higher Correlational design—low- and high- 


participation school comparisons 


Note. In 2017-18 (grant Year 3), the data collection plan was strategically revised to reduce the redundancy of information 
collected over time. Specifically, principal, teacher and IC interviews were removed from the evaluation data collection plan. 
The teacher survey was revised to include questions related to teacher perceptions of principal engagement in teacher PD 
that were originally collected through teacher interview. In this table, additional questions added in the 2017-18 reporting 
period are marked with an asterisk. 


Each research design is discussed as follows. 


Correlational design—single group with repeated measures. SEED was a school-level 
intervention. SEED services were available to all teachers in SEED schools; therefore, it was 
hypothesized that all teachers and students within schools might have benefited from SEED in a 
wide variety of ways, from principal engagement, to SEED course offerings, to SEED PAK 
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resoutces. Therefore, principal, teacher, and student-level data were aggregated to the school level 
and used in the analysis to understand the extent to which the outcomes of interest reported by 
principals, teachers, and students changed over time. Questions for Goals 2a, 2b, 3b, 3c, 3d, 4a, 4b, 
and 6a were addressed using this design with all relevant data collected throughout the project 


period. 


Correlational design—SEED and non-SEED teacher comparisons (non-matched). This 
design was employed to understand the differences in outcomes of interest between SEED and 
non-SEED teachers” utilizing teacher participation data and teacher survey data collected during the 
2018-19” school year (Y3 full implementation). Questions for Goals 2c, 3e and 4c were addressed 
using this design. 


Correlational design—low and high participation schools. This design was employed to 
understand the associations between level of SEED participation at the school level (e.g., low vs. 
high) and the outcomes of interest. Level of SEED participation was measured by the percentage of 
teachers who were defined as SEED participants within each school (see the above “correlational 
design—SEED and non-SEED teacher comparison” section for the definition of SEED 
participation). Data collected from 2018-19 (Y3 full implementation) were used for the analysis. 
Questions for Goals 2d, 3f, 4d, 5e, 5f, and 6b were addressed using this design. 


Quasi-experimental design (QED) with a matched compatison group. A quasi-experimental 
design using propensity score matching methods (PSM) was employed to understand the impact of 
SEED on student achievement (Goals 5a and 5c) and growth (Goals 5b and 5d) over time. The 
primary impact question was: To what extent do SEED schools and comparison schools differ in student 
achievement outcomes after one year, two years, and three years of full implementation (Goal 5a)? To address this 
question, six sub-questions were examined: 


5a1. To what extent do SEED schools and comparison schools differ in reading outcomes 
after one year of full implementation? 


5a2. To what extent do SEED schools and comparison schools differ in reading outcomes 
after two years of full implementation? 


5a3. To what extent do SEED schools and comparison schools differ in reading outcomes 
after three years of full implementation? 


5a4. To what extent do SEED schools and comparison schools differ in math outcomes 
after one year of full implementation? 


5a5. To what extent do SEED schools and comparison schools differ in math outcomes 
after two years of full implementation? 


5a6. To what extent do SEED schools and comparison schools differ in math outcomes 
after three years of full implementation? 


In addition to understanding the impact of SEED on all students within SEED schools, the impact 
on subgroups of students within SEED was also of interest: To what extent do high-needs students from 


'2 The distinction between SEED and non-SEED teachers is described in the Study Samples section. 

13 Due to interruption of data collection by the COVID-19 pandemic, the survey response rates from teachers, principals 
and students during the 2019-20 school year were much lower than the response rates from these groups prior to 2019— 
20. Additionally, the state department of education waived the state standardized assessments during the 2019—20 school 
year. As a result, data collected during the 2018-19 school year had the most complete data to answer the proposed 
evaluation questions. 
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SEED schools and comparison schools differ in student achievement outcomes after one year, two years, and three 
years of full implementation (Goal 5c)? The subgroups included (1) students from racial/ethnic minority 
groups’ (MINORITY) and (2) students receiving free or reduced-price lunch (FRL). The same set 
of six sub-questions described above were also examined for each subgroup as exploratory 
questions. In addition to understand the impact on the outcomes at the end of each year, questions 
related to growth were also examined: To what extent do students form SEED schools demonstrate a 
statistically significant increase in student achievement outcomes over time (Goal 5b)? To what extent do high-needs 
students from SEED schools demonstrate a statistically significant increase in student achievement outcomes over time 


(Goal 5c)? 


In sum, a QED study with six different analytic samples was conducted to address the proposed 
impact questions across student groups (i.e., all students and two different subgroups) and subject 
areas (i.e., ELA and math). They are: 


Sample 1: All students with ELA outcome 
Sample 2: Minority students with ELA outcome 
Sample 3: FRL students with ELA outcome 
Sample 4: All students with Math outcome 
Sample 5: Minority students with Math outcome 
Sample 6: FRL students with Math outcome 


For each QED study sample, PSM was employed to identify a group of matched comparisons. The 
list of covariates in the matching process included baseline data collected during the 2015-16 school 
year (baseline year before full implementation). The covariates include school size, building type, 
locale, percentage of minority students, percentage of students receiving free or reduced lunch, 
percentage of English learners, and school-level student achievement. More information about the 
PSM methods and procedures are described in the Study Samples section. 


Study Samples 


This section discusses the study samples and characteristics of the samples across all evaluation 
questions. 


SEED Schools 


A total of 21 K—12 traditional public schools across seven school districts voluntarily participated in 
the SEED implementation. The 21 participating schools included: (1) nine elementary schools; (2) 
three middle schools; (3) three high schools; (4) two elementary and middle schools; (5) three middle 
and high schools; and (6) one K—12 school. In the 2018-19 school year, one elementary school was 
closed, and most of the students from the school were enrolled in the other two SEED schools 
within the district. Three schools (14%) are in the remote areas, and 18 schools (86%) are in the 
outlying towns or cities where access to high quality PD is often a challenge for educators. Fifteen 
(71%) schools have student enrollment less than 400 (i.e., small schools), and six (29%) schools have 
student enrollment equal or greater than 400. The average percentage of minority students across 
SEED schools was 21%, ranging from 6 to 41% in 2016. The average percentage of FRL students 


4 Racial and ethnic minority was defined as a student with a non-white race/ethnicity. 
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was 38%, ranging from 17 to 61%. The average percentage of EL students was 12%, ranging from 4 
to 26%. Figure A2 shows the distribution of school demographic characteristics and baseline 
achievement data’’ across 21 SEED schools. 


20,00 30.00 


SchSize2016 Minority_pct.1 


Frequency 
Frequency 


10.00 20.00 30.00 40.00 50.00 60.00 70.00 


FRL_pct.1 


00 


ZMath.1 


Figure A3. Distribution of the SEED Schools’ Demographic Characteristics and 
Baseline Achievement Scores 


Because SEED was a school-level intervention, all principals, assistant principals, teachers, and 
students from the participating sites were invited to participate in the evaluation. Table A3 provides 
the number of principals and assistant principals, teachers, and students across 21 SEED 
participating sites by school year based on the school administrative data 

(https: //www.cde.state.co.us /cdereval/staffcurrent). 


15 To allow for cross-grade level comparisons, the within-grade-school mean scores were converted to z-scores using the 
state means and standard deviations calculated from individual student-level data within grade, within test year. See Data 
Analysis Plan section for more information about the methods used to convert scale scores into z-scores. 
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Table A3. Number of Principals, Assistant Principals, Teachers, and Students by 


School Year 


2015-16 


Full Implementation 


Participants (Baseline) 2016-17 2017-2018 2018-19 2019-20 
CET) CET) CE) 

Number of principals 21 21 223 212 212 
(assistant principals) (11) (11) (10) (10) (10) 
Number of teachers 492 495 492 502 500 
(range per school) (12-43) (12-43) (13-44) (12-46) (13-47) 
Number of students 7,069 7180 7348 7320 7179 
(range per school) (84-715) (141-755) (139-827) (133-828) (118-839) 


a One of the participating schools had two principals. 


SEED principals. Within SEED Schools, principals and assistant principals were involved in 
SEED in various ways, such as participating in the principal learning communities and using SEED 
PAK to support teacher PD. 


SEED teachers vs. non-SEED teachers (non-matched). Within SEED Schools, teachers are 
able to participate in SEED in various ways, from taking formal SEED courses to reading a SEED 
blog or completing a SEED PAK resource. Of various types of SEED activities, certain types of 
activities were more likely to have an impact on teacher practices and outcomes, including 


e SEED course participant (individual who participated in at least 80% of the TLC, 
PPS, independent study, and book study) 

e SEED PAK participant (individual who completed at least eight SEED PAK 
resoutces within a school year) 

e SEED actor (individual who reported applying what they learned from SEED in the 
classroom on a weekly or daily basis) 

e SEED facilitator 


When comparing the difference between SEED teachers and non-SEED teachers (1.e., evaluation 
questions that compare the difference between SEED and non-SEED teachers), teachers who met 
at least one of the above criteria were identified as SEED teachers. Teachers who did not meet any 
of the above criteria were identified as non-SEED teachers. 


Non-SEED Matched Comparison Schools 


As mentioned previously, six study samples were analyzed to examine the impact of SEED on 
student achievement outcomes. PSM was employed to identify a group of comparison schools for 
each study sample; the matched comparison schools were drawn from the pool of comparison 
schools across the state of Colorado. Table A4 shows all key covariates that were used in the 
matching for each study sample. However, different variables were created and utilized in different 
study sample, as appropriate, to achieve the best matching results based on the characteristics of 
each unique study sample. As shown in Table A4, for instance, for study sample 1 (S1), the 
covariates used in the matching include SmallSch, MID, HIGH, ELMMID, MIDHIGH, K12, 
Remote, MinorityGP, FRLGP1, FRLGP2, ELGP1, ELGP2, and ELA». For study sample 2 (S2), the 
covariates included in the matching include SmallSch, BuildingGP, Remote, MinorityGP, FRLGP, 
ELGP, and ELAo. 
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Table A4. Baseline Covariates Included in the Matching by Study Sample 


Characteristics Variables Definition Study Samples 
School size SmallSch 0 = student enrollment > 400 
= student enrollment <= x |x |x |x |x |x 
400 
Building type MID, HIGH, Five dummy variables were 
ELMMID, created using elementary 


MIDHIGH, K12 | schools as the reference 


group—MID, HIGH, t onl as zs 
ELMMID, MIDHIGH, and 
K12. 
BuildingGP 0 = elementary schools 
| = middle and/or high x x 
schools 
Locale Remote 0 = non-remote’® 
| = remote ee ee Nee ee lee 
Percentage of MinorityGPI, % Minority was recoded into 
minority students Minority GP2 a categorical variable: 0 = % 


Minority <= 25%; | = % 
Minority between 26 and 
50%; 2 = % Minority >= 51%. 
Two dummy variables were 
created using 0 as the 
reference group. 

MinorityGP % Minority was recoded into 
a categorical variable: 0 = % 
Minority <= 25%; | =% x x x 
Minority between 26 and 
50%. 

Percentage of FRL FRLGPI, % FRL was recoded into a 
students FRL GP2 categorical variable: 0 = % 
FRL <= 25%; | = % FRL 
between 26 and 50%; 2 =% x x x x 
FRL >= 51%. Two dummy 
variables were created using 
0 as the reference group. 
FRLGP % FRL was recoded into a 
categorical variable: 0 = % 
FRL <= 25%; | = % FRL 
between 26 and 50%. 
Percentage of EL ELGPI, % EL was recoded into a 
students ELGP2 categorical variable: 0 = % EL 
<= 10%; | = % EL between 
11 and 20%; 2 =% EL >= 


'6 All SEED schools were in the outlying city, outlying town, or remote areas. Before the matching, schools that were 
not in these types of areas were removed from the dataset. 
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Characteristics WEVa tle) (<1 Definition Study Samples 


21%. Two dummy variables 
were created using 0 as the 
reference group. 

ELGP % EL was recoded into a 
categorical variable: 0 = % EL 


<= 10%; | = % EL between x * 
11 and 20% 
School-mean ELAo/ Matho School mean ELA/math (z- 
student 
scores) for all students and x |x |x |x |x |x 
achievement at 47 
F subgroups at baseline 
baseline 


PSM was conducted using the MatchIt package in R to identify a matched comparison group for 
each of the study samples. For different datasets, different matching algorithms were applied to 
achieve the best matching results. They are: 


e Nearest matching. The nearest matching with a caliper of 0.25 and a matching ratio of 
5 was employed to identify a matched comparison group for Study 1 and Study 2. With 
nearest matching, matching is done using a distance measure specified by the distance 
option. With nearest matching, the SEED schools that did not have similar matches in 
the potential comparison pool were removed from the matching and the analysis. 

e Full matching. CDE masks data for subgroups that have a sample size smaller than 16. 
As a result, the final sample sizes for student subgroups that were retained in the impact 
analyses were quite small. Therefore, for Studies 2, 3, 5, and 6, the full matching 
algorithm was used to minimize a weighted average of the estimated distance measure 
between each treated subject and each control subject within each subclass. To ensure 
the best matching results, schools that did not share the same demographic 
characteristics as the SEED schools were removed from the dataset before matching. 
For instance, for Study 2, none of the SEED schools were categorized under MinGP2, 
FRLGP2, or ELGP2. Therefore, the comparison schools that were categorized under 
MinGP2, FRLGP2, or ELGP2 were removed before matching. This approach was to 
ensure that the comparisons remaining in the dataset were as similar to SEED schools as 
possible before matching. Then, full matching was conducted to generate a weights 
variable to adjust for the baseline difference between the remaining SEED and non- 
SEED schools. With full matching, the remaining SEED and non-SEED schools were 
retained in the dataset, and the weights created by MatchIt was applied in the impact 
analysis to ensure that the treated and control groups were weighted up to be similar. 


After the matching process was complete for each study sample, balance diagnostics were conducted 
to check the quality of the matches (Rubin, 2001). The diagnostic criteria suggested by Rubin (2001) 
below were used to check covariate balances: 


'7 Appropriate baseline achievement data were used in the matching depending on the study sample. For instance, the 
baseline achievement data for all students were used when the matching was conducted with the all-students sample. 
The baseline achievement data for FRL students were used when the matching was conducted with the FRL sample 
only. The baseline achievement data for MINORITY students were used when the matching was conducted with 
racial/ethnic minority (non-white) sample only. 
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e The ratio of the variances of the propensity scores (PS) in the two groups must be close 
to 1.0. Rubin (2001) suggests that the variance ratios should be between 0.5 and 2.0. 

e The difference in the means of the propensity scores in the two groups being compared 
must be small. Rubin (2001) suggests that the standardized differences of means should 
be less than 0.25. 

e For the percent of balance improvement, the larger the percent, the better the PSM 
results. 


Table A5 shows the final sample sizes and the balanced baseline characteristics for the SEED 
schools and comparison schools, as the result of the matching, for each study sample at baseline. 
Specifically, the ratio of the variance of the propensity scores (PS) ranged from 0.53 and 1.38, which 
is within the range recommended by Rubin (2001). The standardized mean differences on all 
covariates, except one, were smaller than 0.25, which also met Rubin’s recommendation. All 
covariates used in the matching were also included in all follow-up analyses and impact analyses to 
account for the leftover imbalance at baseline. The state of baseline equivalence for the samples used 
in the Y1, Y2, and Y3 outcome analyses are reported in Appendix C. 


Table A5. Baseline Characteristics of the Six Study Samples 


Study SEED Schools yarn Variance of the PS Standardized Mean 
SET e) (2) N M SD rrlile) Difference 
All Students with ELA Outcome‘? 

SmallSch 19} 0.68 | 0.48} 81 | 0.66 | 0.48 0.94 0.04 
MID 19 | 0.16 | 0.37) 81 | 0.14 | 0.34 (0.017 / 0.018) 0.07 
HIGH 19 | 0.16 | 0.37 | 81 | 0.15 | 0.36 0.03 
ELMMID 19} 0.11 | 0.32) 81 | 0.12 | 0.33 0.05 
MIDHIGH 19} 0.16 | 0.37) 81 | 0.16 | 0.37 0.01 
K12 19} 0.05 | 0.23 | 81 | 0.03 | 0.18 0.10 
Remote 19} 0.16 | 0.37 | 81 | 0.15 | 0.36 0.03 
MinorityGP 19} 0.26 |0.45) 81 | 0.26 | 0.44 0.01 
FRLGP1 19} 0.58 | 0.51) 81 | 0.54 | 0.50 0.07 
FRLGP2 19} 0.21 | 0.42) 81 | 0.23 | 0.42 0.05 
ELGP1 19} 0.42 | 0.51) 81 | 0.41 | 0.50 0.01 
ELGP2 19} 0.05 | 0.23 | 81 | 0.03 | 0.18 0.10 
ELAo 19 | -0.02 | 0.45) 81 | 0.01 | 0.39 0.06 
SmallSch 7 | 0.14 | 0.38 | 44 | 0.15 | 0.36 1.38 0.03 
BuildingGP 7 | 0.71 | 0.49) 44 | 0.59 | 0.50 (0.058/0.042) 0.24 
FRLGP 7 | 0.43 | 0.53) 44 | 0.44 | 0.50 0.03 
MinorityGP 7 | 0.29 | 0.49) 44 | 0.35 | 0.48 0.13 
ELGP 7 | 0.57 | 0.53) 44 | 0.68 | 0.47 0.22 


'8 Two SEED schools were removed from the study sample due to lack of matched comparisons from the potential 
comparison schools. 


' The variable Remote was not included in the matching because none of the SEED schools remaining in the final 
sample were in remote areas. Therefore, other non-SEED schools in remote areas were also removed from the dataset 
prior to matching. 
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Non-SEED 


SEED Schools S Variance of the PS Standardized Mean 
eee) i) : iff 
N M SDN M SD ratio Difference 
ELAo 7 | -0.21 | 0.36) 44 | -0.23 | 0.24 0.08 
SmallSch 15 | 0.60 | 0.51 | 159 | 0.71 | 0.46 0.53 0.24 
MID 15 | 0.20 | 0.41 | 159) 0.27 | 0.44 (0.023 / 0.043) 0.16 
HIGH 15} 0.13 | 0.35) 159] 0.13 | 0.33 0.02 
ELMMID 15 | 0.13 | 0.35 | 159 | 0.09 | 0.28 0.16 
MIDHIGH 15 | 0.13 | 0.35) 159) 0.10 | 0.30 0.12 
Remote 15 | 0.07 | 0.26 | 159 | 0.07 | 0.25 0.01 
MinorityGP 15 | 0.33 | 0.49 | 159 | 0.42 | 0.50 0.17 
FRLGP' 15 | 0.60 | 0.51 | 159 | 0.64 | 0.48 0.09 
FRLGP2 15 | 0.13 | 0.35) 159 | 0.13 | 0.34 0.01 
ELGP1 15 | 0.47 | 0.52 | 159 | 0.49 | 0.50 0.04 
ELGP2 15 | 0.07 | 0.26 | 159 | 0.07 | 0.26 0.01 
ELAo 15 | -0.40 | 0.29 | 159 | -0.35 | 0.21 0.19 
4 All Students with Math Outcome 2 
SmallSch 18 | 0.67 | 0.49) 72 | 0.77 | 0.43 0.94 0.23 
MID 18 | 0.17 | 0.38) 72 | 0.15 | 0.36 (0.33 / 0.35) 0.04 
HIGH 18 | 0.17 | 0.38) 72 | 0.16 | 0.37 0.02 
ELMMID 18 | 0.11 | 0.32 | 72 | 0.10 | 0.31 0.02 
MIDHIGH 18 | 0.17 | 0.38 | 72 | 0.20 | 0.41 0.09 
K12 18 | 0.00 | 0.00) 72 | 0.03 | 0.17 0.19 
Remote 18 | 0.11 | 0.32) 72 | 0.15 | 0.36 0.11 
MinGP1 18 | 0.28 | 0.46} 72 | 0.26 | 0.44 0.04 
MinGP2 18 | 0.00 | 0.00) 72 | 0.03 | 0.18 0.21 
FRLGP' 18 | 0.61 | 0.50) 72 | 0.64 | 0.48 0.05 
FRLGP2 18 | 0.17 | 0.38) 72 | 0.20 | 0.40 0.07 
ELGP1 18 | 0.39 | 0.50) 72 | 0.33 | 0.47 0.12 
ELGP2 18 | 0.06 | 0.24) 72 | 0.04 | 0.21 0.05 
Matho 18 | -0.01 | 0.41 | 72 | -0.04 | 0.40 0.08 
5 Minority Students with Math Outcome 
SmallSch 7 | 0.14 | 0.38) 51 | 0.10 | 0.30 0.98 0.15 
BuildingGP 7 | 0.71 | 0.49} 51 | 0.77 | 0.42 (0.47 / 0.48) 0.13 
MinorityGP 7 | 0.43 | 0.53) 51 | 0.52 | 0.50 0.18 
FRLGP 7 | 0.29 | 0.49) 51 | 0.32 | 0.47 0.08 
ELGP 7 | 0.57 | 0.53) 51 | 0.44 | 0.50 0.27* 
Matho 7 | -0.15 | 0.37) 51 | -0.16 | 0.23 0.04 
6 FRL Students with Math Outcome 
SmallSch 15 | 0.60 | 0.51 | 257 | 0.56 | 0.50 1.00 0.07 
MID 15 | 0.20 | 0.41 | 257) 0.24 | 0.43 (0.34 / 0.34) 0.08 
HIGH 15 | 0.13 | 0.35 | 257 | 0.15 | 0.36 0.06 
ELMMID 15 | 0.13 | 0.35 | 257 | 0.13 | 0.34 0.00 
MIDHIGH 15 | 0.13 | 0.35 | 257] 0.11 | 0.31 0.08 


?0'Three SEED schools were removed from the study sample due to lack of matched comparisons from the potential 


comparison schools 
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IN Co) gets) = =D) 


SEED Schools Senools 

M SD N M SD 
Remote 15 | 0.07 | 0.26 | 257 | 0.06 | 0.24 
MinGP1 15 | 0.33 | 0.49 | 257 | 0.29 | 0.45 
MinGP2 15 | 0.00 | 0.00 | 257 | 0.03 | 0.17 
FRLGP1 15 | 0.60 | 0.51 | 257 | 0.59 | 0.49 
FRLGP2 15} 0.13 | 0.35) 257 | 0.14 | 0.35 
ELGP1 15 | 0.47 | 0.52 | 257 | 0.42 | 0.50 
ELGP2 15 | 0.07 | 0.26 | 257 | 0.09 | 0.29 
Matho 15 | -0.44 | 0.30 | 257 | -0.39 | 0.28 


Variance of the PS 


ratio 


Standardized Mean 


Difference 


0.03 


0.10 


0.18 


0.03 


0.03 


0.09 


0.09 


0.21 


* The standardized mean difference on the variable was greater than 0.25, suggesting that the SEED and non- 
SEED schools were different on the percentage of EL students served at baseline after matching and adjusting for 
weights. In other words, there was a much higher percentage of EL students in the SEED schools compared to 
non-SEED schools. 


Data Collection Methods 


This section describes all data collection conducted throughout the project period, including the 
pilot testing and baseline data collection year (2015-16) as well as four years of full implementation 
(2016-2020). Institutional review board (IRB) approval was obtained in 2015 before school 
recruitment. All participating principals and teachers provided consent online to participate in the 
evaluation. Documentation of student parental consent was waived, but parents could choose to opt 
their children out of the survey. Throughout the project period, instruments were revised as needed 
based on quality and informativeness of the data. The teacher survey, in particular, was revised 
annually to reflect changes made to SEED offerings. Table A6 shows the data collection timeline 
and instruments that were used throughout the full implementation period. 


Table A6é. Data Collection Timeline Throughout the Project Period 


Pilot Sample Full Implementation 
Instrument 2015 2016 AUT) 2017 2018 2019 2020 
(Baseline) (Follow-Up) (Baseline) (Y1) (¥2) (Y3)  (¥4) 

Fidelity of Implementation Matrix X X X X X 
Principal baseline interview X 
Principal follow-up interview X X 
Teacher baseline interview X X 
Teacher follow-up interview X X 
IC baseline interview X 
IC follow-up interview X X 
Principal baseline survey X 
Principal follow-up survey X X X X X 
Teacher baseline survey X X 
Teacher follow-up survey X X X X X 
Student engagement survey (3'-5'"") X X X X X 
Student engagement survey (6-10!) X X X X X 


Note. All data collection occurred annually throughout the project period except the 2015 baseline data collection, which 
occurred in November/December 2015. 
All = all sample, T1 = time 1 assessment, T2 = time 2 assessment, T3 = time 3 assessment, T4 = time 4 assessment. 
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All instruments were developed and tested during the pilot year (2015-16), except the student 
engagement surveys, which were first administered at the baseline of full implementation (May/June 
2016). The final implementation year was the 2019-20 school year; however, the implementation 
was disrupted by the COVID-19 pandemic in March 2020. All participating schools were closed for 
the remainder of the school year. NW BOCES continued their PD delivery completely online during 
school closures. McREL administered the teacher survey, principal survey, and student survey 
online, as planned. However, due to school closures, the state assessments and activities related to 
teacher evaluations were suspended. As a result, several key data elements for the final evaluation 
were unavailable, including student achievement data and teacher evaluation ratings. Additionally, 
due to the disruption caused by COVID-19, survey response rates across all participant groups were 
lower than for previous administration years. Each instrument and the corresponding response rates 
are briefly described in the following sections. 


Fidelity Assessment 


McREL evaluators developed a fidelity of implementation (FOT) matrix to examine and monitor the 
level of implementation throughout the project period. As shown in Table A7, eight fidelity 
indicators were developed to measure all four SEED strategies across three main categories: SEED 
creation, SEED delivery, and SEED participation. Indicators of SEED creation and SEED delivery 
were measured at the program level. Indicator scores were combined to create a program-level score 
for each category, and a threshold for implementation with fidelity at the program level was defined 
as 3. SEED participation indicators were measured at the teacher level, rolled up to the school level, 
and then rolled up to the program level. A threshold for implementation with fidelity at the 
program-level was also defined as 3. Various data sources were used to assess each of the specified 
fidelity indicators. See Project Record and Extant Data section for more details regarding each of the 
data sources. 


The FOI matrix contains a quantitative system for rating each key component of SEED. Level of 
implementation serves as the scoring system for the adherence index (low to high; 1 to 3). A mean 
score was determined for each stage of the project (initial planning, technology development, 
content development, phased-in implementation, full implementation, and sustainability). Means 
across all key components were calculated to determine a “score” for the dosage index. For example, 
each teacher had data indicating whether they had SEED PD linked to specific teacher performance 
indicators. The percentages of teachers at each school who met this criterion (yes or no) were 
averaged across all schools to get a “score” for this SEED participation key component for the 
entire sample. The FOI matrix was pilot tested with the pilot teacher and principal samples in 2015— 
1G; 
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Table A7. SEED Fidelity of Implementation Matrix 

aT OE a eee a 
next higher Roll-up to 

level if program level 
(score and 


pee threshold for 
adequate 


Threshold for 
adequate 
implementation 
at unit level 


Expected 
sample for 
fidelity 
measure 


Expected 
years of 
fidelity 
measurement 


Score for levels of 
Collection implementation at unit 
(who, when) level 


Unit of Data 


Indicators Definition 


implementation Source(s) 


threshold): 


Indicate 
level 


implementation 
at sample level) 


. Topics for Learning Program PD committee | By June, Percent of learning 3 NIA Percent of Program Years 1-4 
learning strands and content meeting strands that meet learning strands (2016-2020) 
strands for chosen development | minutes put | criteria. Score based on that meet criteria. 

PD are based on team (CDT) on Google percent: (1) low; (2) Score based on 
selected teacher meeting Drive med; (3) high. percent: (1) low; 
based on need minutes (2) med; (3) high. 
teacher Threshold=3. 
needs 
2. Evidence- PD activity | PD activity SEED PAK By Sept., Score assigned to each | 1 % of 20 % of 20 randomly | SEED PAK Years 1-4 
basedPDis | has resources McREL activity: randomly selected SEED resources (2016-2020) 
selected by theoretical reviewed to reviews PD | 0 = no indication of selected PAK resources throughout 
ICs and evidence determine against theoretical base; and 1 SEED PAK with score of 1. the project 
CDTs for base whether they | research- = identified theoretical resources Score based on 
inclusion in behind it contain based theory | base with score of | percent: (1) low; 
the SEED strategies that 1. Score (2) med; (3) high. 
PAK are supported based on Threshold = 3. 
by research- percent: (1) 
based low; (2) med; 
theory?" (3) high. 
Threshold = 3 
3.1Cs and TLCs Program ICs’ syllabi for | By Sept. and | Percent of TLCs that 3 NIA Percent of TLCs | Program Years 1-4 
CDTs create | include at the TLCs Feb., SEED | include at least 2 in- that include at (2016-2020) 
in-person least two TLC syllabus | person activities per least 2 in-person 
TLC activities | in-person put on semester. Score based activities per 
activities Google Drive | on percent: (1) low; (2) semester. Score 
per med; (3) high. based on 
semester percent: (1) low; 
(2) med; (3) high. 
Threshold = 3 
4.1Cs and TLCs that Program ICs’ syllabi for | By Sept. and | Percent of TLCs that 3 N/A Percent of TLCs | TLCs Years 1-4 
CDTs create | have at the TLCs Feb., TLC have a least one hour that have a least (2016-2020) 
least one syllabus per week of virtual one hour per 


21'The McREL evaluation team randomly selected 20 PD resources from SEED PAK annually to determine whether the resources are backed up by the evidence-based 


literature. 
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Roll-up to 
next higher 
Threshold for level if 
adequate needed 
implementation (score and 
at unit level Miccire)(¢) 8 
Indicate 
level 


Roll-up to 
program level 
(score and 
threshold for 
adequate 
implementation 
at sample level) 


Expected Expected 
sample for years of 
fidelity fidelity 
measure measurement 


Data Score for levels of 
Collection implementation at unit 
(who, when) level 


Unit of Data 


Indicators Definition implementation Source(s) 


virtual TLC hour per posted on activities. Score based week of virtual 
activities week of Google Drive | on percent: (1) low; (2) activities. Score 
virtual med; (3) high. based on 
activities percent: (1) low; 
(2) med; (3) high. 
Threshold = 3 
5. Cs format TLCs utilize | Program TLC reviewed | Feb. and Percent of TLCs that 3 N/A Percent of TLCs | Program Years 1-4 
TLC delivery | the four against Sept., utilize the four evidence- that utilize the (2016-2020) 
using evidence- Vescio et al. McREL based TLC standards. four evidence- 
evidence- based TLC (2008) TLC reviews each | Score based on based TLC 
based standards standards 22 TLC’s percent: (1) low; (2) standards. Score 
delivery from Vescio syllabus put | med; (3) high. based on 
standards et al. on the percent: (1) low; 
(2008) Google Drive (2) med; (3) high. 
model. Threshold = 3. 
Total Score for SEED PD Creation Components Mean of the five indicators. Threshold = 3. 
6.ICs facilitate | TLC is Program ICs’ syllabi for | By Aug. and | Percent of learning 3 NIA Percent of Program Years 1-4 
blended hosted for TLCs and Jan., strands that have TLCs learning strands (2016-2020) 
learning each content documents hosted as indicated in that have TLCs 
TLCs learning development | are put on syllabi. Score based on hosted as 
strand team meeting | Google Drive | percent: (1) low; (2) indicated in 
minutes med; (3) high. syllabi. Score 
determining based on 
learning percent: (1) low; 
strands (2) med; (3) high. 


Threshold=3. 
Total Score for SEED PD Delivery Component Same as indicator 6. Threshold = 3. 


7. SEED PDis | Principal or | Teachers SEED system | By June, 0=No 1 School level: | Percent of All schools in | Years 1-4 
linked to teacher linking SEED 1= Yes % of schools that have | which the (2016-2020) 
specific assigns at COPMS with | COPMS data teachers with | score of 2. Score | intervention 
teacher least one will be score of 1in | based on is being 
performance | SEED PD provided to each school. | percent: (1) low; | implemented 
indicators by | in COPMS McREL Score based | (2) med; (3) high. 

on percent: Threshold=3. 
(1) low (0- 


2 ICs created a syllabus for each TLC. The McREL evaluation team reviewed TLC syllabi and compared the syllabi against the evidence-based professional/ teacher learning 
community practices as identified by Vescio et al. (2008). 
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Roll-up to 


: Roll-up to 
eee te) rogram level 
Threshold for level if prog Expected Expected 
Unit of Data po Ped bl IoC adequate needed Geo ane sample for years of 
Indicators Definition . ; Collection implementation at unit . F threshold for ae a 
implementation Source(s) implementation (score and Lief fidelity 
ine wien) ie at unit level threshold): poeduae measure measurement 
: * implementation 
ate Cer} (2) 
level at sample level) 
teachers or SEED 19%); (2) 
principals platform24 med (20- 
50%); (3) 
high (51- 
100%). 
Threshold=2 
8. Teachers Teacher Teachers SEED By June, 0=No 1 School level: | Percent of All schools in | Years 1-4 
complete completes platform PD SEED 1= Yes % of schools that have | which the (2016-2020) 
SEED PD# | atleast one completion platform data teachers with | score of 2. Score | intervention 
facilitated data and will be score of 1in | based on is being 
SEED PD Innovation provided to each school. | percent: (1) low; | implemented 
OR self- Coach TLC McREL and Score based | (2) med; (3) high. 
directed gradebook by Jan. and on percent: Threshold = 3. 
completion data2¢ and June, IC (1) low (0- 
of at least 8 other “gradebook” 19%); (2) 
SEED PAK facilitated use | and med (20- 
activities (participation | facilitated 50%); (3) 
per score must be | use data will high (51- 
academic atleast 80% | be uploaded 100%). 
year to be on Google Threshold = 
“complete” Drive. 2 


Mean of the two indicators. Threshold = 2. 


Total Score for SEED Participation Component 
Note. Unless otherwise noted, when a score was assigned based on percent, 1= low (0-33%); 2 = med (34-66%); 3 = high (67—100%). 


3 This indicator is measured by the number of teachers who were assigned, self-assigned, or assigned by others, with at least one SEED PAK resource during the school year. 
* This is a digital system that links the Colorado Performance Management System (COPMS) with SEED. Annually, RANDA Solutions provided the data to determine 

whether teachers have SEED PAK activities connected to their professional growth plans in the COPMS. 
25 SEED PD included SEED PD offerings (i.e., TLCs, PPS, SEED PAK independent study, book study, and others [i.e., Alt-licensure, induction, EL modules, SEED 
facilitator]) and SEED PAK (..e., complete at least 8 resources within one school year). 


6 TCs were responsible for tracking teacher participation in SEED offerings; these data were used to determine teacher completion status across SEED offerings. 
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Principal Baseline and Follow-Up Interviews 


Two types of principal interview protocols were developed: baseline and follow-up. The baseline 
principal interview protocol was developed in fall 2015 and was designed to understand current 
systems or processes in place at the school and district levels to support principal involvement in 
teacher professional growth, what principals perceived their roles to be regarding teacher 
professional growth, and whether they were satisfied with their roles and why or why not. All 21 
school principals were interviewed. 


The principal follow-up interview protocol was developed in spring 2016 (pilot follow-up/baseline 
full implementation) and was designed to understand principals’ current roles in and current 
practices for teacher professional growth, any changes in beliefs or practices since SEED was 
implemented during the pilot, recommendations for changes to SEED, and success with SEED. All 
21 principals from the SEED schools were interviewed again using the principal follow-up interview 
protocol. 


The protocol was revised in spring 2017 (Y1 full implementation) by adding one extra question 
about what SEED meant to them (i.e., principals). A stratified sample of eight principals were 
interviewed using the slightly revised principal follow-up interview protocol. The principal selection 
process was based on each school’s fidelity score for SEED participation. Within each quartile, two 
principals were selected for interview. When there were different grade levels in the quartile, one 
principal from each grade level (primary and secondary) was purposefully selected. During the 2017 
interview, there was no primary school in the 3" quartile, so the 4° quartile was oversampled. 
Additionally, one principal within the 3" quartile did not respond to the interview invitation; hence, 
another principal within the same quartile with the nearest percentage was selected and completed 
the interview. 


All interviews were conducted over the phone. Each interview was about 30 minutes during each 
round of data collection. Starting in 2017-18, principal interviews were removed from the data 
collection to reduce the redundancy of information collected over time. 


Innovation Coach (IC) Interviews 


An IC interview protocol was developed during the pilot baseline to understand IC perceptions of 
their readiness to support teachers; whether SEED infrastructure was adequate to support teachers; 
how SEED supported their work with teachers; successes and barriers with teachers; and barriers 
with SEED. The same protocol was used in May 2016 and 2017 (baseline and Y1 full 
implementation). All interviews were conducted over the phone and each interview was about 30 
minutes. Starting in 2017-18, IC interviews were removed from the data collection to reduce the 
redundancy of information collected over time. 


Teacher Baseline and Follow-Up Interviews 


Two types of teacher interview protocols were developed: baseline and follow-up. At baseline in fall 
2015, the teacher interview protocol was designed to understand current systems or processes in 
place at the school and district levels to support principal involvement in teacher professional 
growth, what teachers would like principals’ roles to be in their professional growth and what their 
roles are, whether they were satisfied with principals’ roles, and why or why not. In addition, 
teachers were asked to describe their current practices in their own professional growth and how 
they chose PD opportunities and strategies. A stratified sample of 12 pilot teachers was interviewed 
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using the teacher baseline interview protocol. The same protocol was used again in spring 2016 to 
collect baseline data for full implementation from the non-pilot teachers. Another stratified sample 
of 12 teachers was interviewed using the baseline protocol. 


The teacher follow-up interview protocol was developed in spring 2017 (Y1 full implementation). 
The interview protocol was designed to understand teachers’ perceptions of their principal’s current 
role in and practices for teacher professional growth; any changes in principals’ and their own beliefs 
of practices since SEED was implemented; any changes in teachers’ own approach to professional 
growth, beliefs about what it means to be a successful teacher, and practice in classroom since 
SEED was implemented; any areas of SEED that could be improved; and success with SEED. A 
stratified sample of 12 teachers (including pilot and non-pilot teachers) was interviewed using a 
slightly revised teacher follow-up interview protocol. Each interview was about 30 minutes during 
each round of data collection. All interviews were conducted over the phone. Starting with 2017-18, 
teacher interviews were removed from the data collection to reduce the redundancy of information 
collected over time. 


In terms of the stratification strategy during the pilot, teacher samples were stratified based on 
current data available from CDE: the school’s highest grade served (primary or secondary), number 
of students (median split), and percentage of students who scored at least proficient on the TCAP 
assessment (median split). The percentage of the 12 teachers selected for each stratum was based on 
the percentage of schools that fell into each stratum (e.g., 19% of schools fell into a stratum, which 
indicated two of the 12 teachers should be sampled). Starting in the 2017 data collection phase, the 
selection process was changed to gather information from a representative sample of TLC and PPS 
participants. Specifically, each teacher was to be assigned a score based on their level of 
participation, and the plan was to use the participation score to select eight teachers from the 
TLC/PPS users stratified by participation score. Four additional teachers from the non-TLC users 
(varying levels of SEED PAK use) were also to be selected. However, for the 2017 teacher sample 
selection, the process was revised slightly. Specifically, based on the participation data, only two 
teachers completed activities on the SEED PAK; hence, these two teachers were invited to 
participate in an interview. Also, based on the participation scores, there were only three values for 
quartiles, so the 10 remaining teachers were selected across the three scores, with four teachers being 
from the largest quartile and three teachers from the other two quartiles. When there were different 
grade levels in the quartile, one teacher from each grade level (primary and secondary) was 
purposefully selected. 


All interviews were conducted over the phone. Each interview was about 30 minutes during each 
round of data collection. Starting in 2017-18, principal interviews were removed from the data 
collection to reduce the redundancy of information collected over time. 


Principal Baseline and Follow-Up Surveys 


Two types of surveys were developed for principals and assistant principals: baseline and follow-up. 
In the baseline principal survey developed in fall 2015, the questions were designed to understand 
principals’ perceptions of school approach to PD. Specifically, 12 items adapted from the 2014 
North Carolina Teacher Working Conditions survey (New Teacher Center, 2014) and 4 items 
developed by McREL were included in the survey. All items were rated on a 1 to 6 (strongly disagree to 
strongly agree) scale. Exploratory factor analysis (EFA) was conducted to identify the underlying 
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constructs of the 16 items used to measure school approach to PD.”’ However, due to the small 
sample size (28 principals and assistant principals responded to the survey at baseline in fall 2015), 
results from EFA may not be reliable (Comrey & Lee, 1992). Therefore, McREL researchers utilized 
the findings from EFA as a supplemental resource to guide the identification of the constructs and 
items that are appropriate to measure the constructs. 


The principal follow-up survey was developed in May 2016 (baseline full implementation) and used 
again annually throughout the implementation period. In addition to the 16 questions about school 
approach to PD, additional items were included—specifically, 6 items to assess principal perception 
of SEED’s utility to support teacher PD (see Table A6)* and 3 open-ended questions asking about 
areas in need of improvement, success with SEED, and additional PD offered to teachers in the 
current school year. 


Principal surveys were administered online with all SEED principals and assistant principals 
throughout the project period. Table A8 shows the response rate and survey length by data 
collection phase. Because SEED is a school-level intervention, all principals were exposed to SEED 
beginning in fall 2015; hence, the 2015 principal baseline survey data served to understand how 
SEED changed principals’ perceptions and practices regarding teacher PD. The low response rate 
during the 2020 follow-up survey may be attributed to the disruption caused by the 2020 COVID-19 
pandemic. 


Table A&8. Principal Survey Response Rate by Data Collection Phase 


Data Collection Phase | Number Invited Number Responded Response Rate Survey Length 


2015 Principal Baseline Survey 32 30 93.8% 10 minutes 
2016 Principal Follow-Up Survey 32 24 15% 15 minutes 
2017 Principal Follow-Up Survey 32 32 100% 15 minutes 
2018 Principal Follow-Up Survey 32 32 100% 15 minutes 
2019 Principal Follow-Up Survey 31 29 93.5% 15 minutes 
2020 Principal Follow-Up Survey 31 18 58.1% 15 minutes 


Table A9 shows the items by construct measured in the principal surveys, including principal 
baseline survey (PB) and principal follow-up survey (PF). These constructs were the outcomes of 
interest measured in the principal surveys. 


Table A9. Constructs Measured in Principal Surveys 


Constructs 
(Data Sources) 
School offering evidence-based | e The PD my school offered enhanced teachers’ ability to implement 
PD (PB, PF) instructional strategies that meet diverse student learning needs. 
e The PD my school offered provided ongoing opportunities for teachers to 
work with colleagues to refine teaching practices. 
e The PD my school offered enhanced teachers’ abilities to improve student 
learning. 


Items 


27 Principal axis factor analysis with varimax rotation was conducted. Based on the scree plot, two-factor and three- 
factor solutions were requested. Interpretation of the results in light of the conceptual framework suggests three-factor 
solution. The first factor accounted for 45% of the variance, the second factor accounted for 11%, and the third factor 
accounted for 6%, with cumulative variance of 62%. 

8 Items were measured on a 1 to 6 (strongly disagree to strongly agree) scale. 
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Constructs 
(Data Sources) 


Items 


The professional learning opportunities my school offered were aligned 
with the school’s improvement plan. 

The PD my school offered deepened teachers’ content knowledge. 

My school’s PD offerings were data-driven. 

| recommended and planned PD based on needs identified through the 
teacher evaluation process. 


School-provided time and 
resources to access PD 
(PB, PF) 


An appropriate amount of time was provided for PD at my school. 
Sufficient resources were available for PD in my school. 


Principal Engagement in 
Teacher PD (PB, PF) 


In this school, follow-up was provided from PD. 

The PD my school offered was evaluated and results were communicated 
to teachers. 

Teachers were encouraged to reflect on their own practice. 

The current teacher evaluation systems provide information that allows me 
to be supportive of teacher PD. 

| understand what PD support teachers need. 

During conversations with teachers regarding their professional growth 
(i.e., improvement on the teacher evaluation rubric and in knowledge and 
implementation of best practices), | feel prepared to be supportive. 


School-offered individualized 
teacher PD (PB, PF) 


The PD my school offered was differentiated to meet the individual needs 
of teachers. 


SEED utility to support teacher 
POE) 


SEED content is useful for teachers’ PD. 

The SEED system has PD that is relevant to teachers’ needs as identified 
by the state evaluation rubric (RANDA). 

SEED PD enabled me to support teachers’ professional growth. 

The integration of the SEED PAK and RANDA is easy to use. 

The integration of the SEED PAK and RANDA allowed me to better track 
teacher professional growth plans. 

The integration of the SEED PAK and RANDA helped facilitate 
conversations with teachers about professional growth. 


Table A10 shows Cronbach’s alphas for the constructs measured in the principal survey over time. 
Overall, reliability estimates were acceptable across all scales over time (a > = 0.70) with one 
exception—the internal consistency of the items measuring principal engagement in teacher PD was 


0.61 in Y4. 


Table A10. Reliability Estimates Over Time: Constructs Measured in the Principal 


Surveys 


Constructs 


Pilot 


Full Implementation 
Baseline Y1 Y2 Y3 Y4 


Baseline 


PA) 


(spring (spring (spring (spring (spring 
PANT) 2017) PAUHT:)) AUK) ) yAIYA\)) 


nl PAN mo hel 


Se offering evidence-based | 58 | ¢ 99 | 24 | 0.78 | 31 | 0.93 | 32 | 0.80 | 28 | 0.92 | 18 | 0.85 
School-provided time and 29 | 0.92 | 24 | 0.91 | 32 | 0.96 | 32 | 0.87 | 29 | 0.94 | 18 | 0.81 
resources to access PD# 
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Pilot Full Implementation 


Baseline Baseline Y1 Y2 Y3 Y4 
Constructs (spring (spring (spring (Spring (Spring 
2016) 2017) 2018) AK) 
Principal engagement in 
Teacher PD 29 | 0.83 | 24 | 0.76 | 30 | 0.81 | 32 | 0.75 | 28 | 0.71 | 18 | 0.61 
eet es | fe || 088 || ae | oes ey | oe | a7 |) oe | i | ee 


@ The construct includes two items. Hence, a was not conducted; rather the value was the correlation between these two items. 
> This construct includes one item; hence, a was not calculated. 
Note. Constructs measured by one item are not reported in this table. 


Teacher Baseline and Follow-Up Surveys 


Two types of surveys were developed for teachers: baseline and follow-up. The baseline teacher 
survey developed in the fall of 2015 was designed to understand school approach to PD; access to 
evidence-based, up-to-date content; and opportunities to collaborate with job-alike peers. To 
measure school approach to PD, 12 items adapted from the 2014 North Carolina Teacher Working 
Conditions survey (New Teacher Center, 2014), 7 items adapted from the Teacher Perceptions of 
Professional Learning survey (Yates & Harris, 2003), and 1 item developed by McREL were 
included. Factor analysis was conducted to identify the underlying constructs explaining teacher 
perception of school approach to PD using the teacher survey data collected at baseline for the pilot 
and baseline for the full implementation.” McREL also developed two items to assess the extent to 
which teachers agreed they had access to evidence-based and up-to-date PD content through 
district, BOCES, or school initiatives. All items were rated on a 1 to 6 (strongly disagree to strongly agree) 
scale. Opportunities to collaborate with job-alike peers was assessed with a single item ona 1 to 5 
(daily, weekly, monthly, less than once/ month, never) scale, followed by an additional item asking how the 
oppottunities are organized (formally organized by administrators, PD programs, and/or teachers, and/or 
informal meetings/ conversations among teachers). The same baseline survey was administered with the non- 
pilot teachers in May/June 2016 to gather the baseline data before full implementation. 


The follow-up teacher survey was developed and administered with the pilot teachers in May 2016. 
The survey contained the same set of items from the baseline survey, including school approach to 
PD (20 items), access to evidence-based and up-to-date content items (2 items), and opportunities to 
collaborate with job-like peers (1 item). Additional items were included to (1) understand SEED’s 
impact on teacher instruction and student learning (6 items); (2) teachers’ PD (6 items); (3) teachers’ 
experience with SEED components (1e., TLC [3 items], IC [4 items], SEED 

Personalized Accessible Knowledge (PAK) [7 items]), and (4) SEED impact on teacher 
collaboration with peers (4 items).*’ All items were rated on a 1 to 6 (strongly disagree to strongly agree) 
scale. Two open-ended questions asking about areas in need of improvement and success with 
SEED were also included. 


°° Principal axis factor analysis with promax rotation was conducted. Based on the scree plot, three-factor and four- 
factor solutions were requested. Interpretation of the results in light of the conceptual framework suggests four-factor 
solution. The first factor accounted for 62% of the variance, the second factor accounted for 7%, the third factor 
accounted for 3%, and the fourth factor accounted for 2%, with cumulative variance of 73%. 

30 Ttems measuring SEED impact on teacher collaboration with peers were adapted from the Teacher Perceptions of 
Professional Learning survey (3 items) (Yates & Harris, 2003) and the Professional Development Activity Log (1 item) 
(Yoon, Garet, Birman, & Jacobson, 2006). 
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The follow-up survey was revised in May 2017. Specifically, revisions were made to the items that 
assess SEED’s impact on teacher instruction and student learning, SEED’s impact on teachers’ PD, 
and teachers’ experience with SEED components, to reflect changes made to the structure and 
definitions of SEED components during the full implementation. Additionally, three questions 
assessing teachers’ growth mindset were added because ICs were recognizing changes in teacher 
mindset. These questions were adapted from the PATH Math Teacher Survey (Yaeger, 2017). In 
May 2018, the teacher follow-up survey was revised again to add items measuring teacher perception 
of the quality of SEED PAK and the SEED blog, as well as teacher perception of principal 
engagement in PD. The teacher survey also collected data on their perceptions regarding the extent 
to which PD content is up-to-date, PD content is evidence-based, and how frequently they apply 
what they have learned from SEED in their classrooms. Each of these constructs was measured by 
one item. 


Table A11 shows the response rate and survey length by data collection phase. All teacher surveys 
were administered online. The numbers of teachers who were invited to take the survey were 
provided by the SEED staff; hence, these numbers were slightly different from the administrative 
data on the CDE website. The low response rate during the 2020 follow-up survey may be attributed 
to the disruption caused by the 2020 COVID-19 pandemic. 


Table All. Teacher Survey Sample, Response Rate, and Survey Length by Data 
Collection Phase 


| DataCollection © Sample Number = Number | Response  SurveyLength | 
Phase Invited Responded Rate (minutes) 
ee eee! Pilot teachers 55 42 76.4% 15 
Baseline Survey 
2016 Teacher Follow: | piiotteachers | 55 37 67.3% 15 
Up Survey 
2016 Teacher Non-pilot 477 296 62.1% 15 
Baseline Survey teachers 
Sea eee WOON) NeAanenctorea | e507: 281 55.4% 15 
Up Survey 
2018 Teacher Folow- | piteachers | 525 377 71.8% 15 
Up Survey 
2019 Teacher Follow: | i'teachers | 545 414 76% 15 
Up Survey 
2020 Teacher Follow” |) wiiteachers | 547 293 53.6% 15 
Up Survey 


aThis number is provided by NW BOCES, which is different from the data published on CDE website. For the calculation of the 
response rate, NW BOCES’s record is used. 


Table A12 shows the items by construct measured in the teacher surveys, including teacher baseline 
survey (IB) and teacher follow-up survey (TF). 
Table Al 2. Constructs Measured in Teacher Surveys 


Constructs 


(Data Sources) 
Instructional practice- 
focused PD (TB, TF) 


| learned new and different ideas from my PD experiences. 
The PD improved my knowledge of instructional strategies. 
Knowledge gained from the PD improved my teaching skills. 
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Constructs 
(Data Sources) 


Items 


e The PD increased my enthusiasm for teaching. 
e The PD encouraged me to reflect on aspects of my teaching. 
e The PD gave me useful ideas of how to improve student outcomes. 
e The PD updated my professional knowledge. 
e The PD | was offered deepened my content knowledge. 
e The PD | was offered was differentiated to meet my individual needs. 
Data-driven PD e The professional learning opportunities | was offered were aligned with my school’s 
(TB, TF) improvement plan. 
e The PD | was offered was data-driven. 
e Teachers were encouraged to reflect on their own practice. 
e School leadership recommended and planned PD based on needs identified through 
the teacher evaluation process. 
e In my school, follow-up was provided from PD. 
e The PD my school offered was evaluated and results were communicated to 
teachers. 
School-provided time | ¢ An appropriate amount of time was provided for PD at my school. 
and resources to e Sufficient resources were available for PD in my school. 
access PD 
(TB, TF) 
Student learning- e The PD | was offered enhanced my ability to implement instructional strategies that 
focused PD meet diverse student learning needs. 
(TB, TF) e The PD | was offered provided ongoing opportunities for me to work with colleagues 
to refine teaching practices. 
e The PD | was offered enhanced my abilities to improve student learning. 
Up-to-date PD e PD content is up-to-date. 
(TB, TF) 
Evidence-based PD |e PDcontent is evidence-based. 
(TB, TF) 
Opportunity to e Onaverage, to what extent do you have opportunities to collaborate with job-alike 
collaborate peers? 
(TB, TF) 
Growth mindset e Some people are just born great teachers; some people are just born great teachers; 
(TF) if you're not, there’s not much you can do to become a really great teacher. 
e If | really try hard, | can get even the most difficult or unmotivated student to learn. 
e | canimprove my instruction through my efforts. 
Quality of SEED PAK | e SEED PAK activities were a good use of my time. 
(TF) e SEED PAK has useful activities and content. 
e SEED PAK increased my knowledge of best practices. 
Quality of SEED blog | e Reading the SEED blog is a good use of my time. 
(TF) e The SEED blog has useful information and content. 
e The SEED blog increased my knowledge of best practices. 
Quality of SEED e SEED courses were a good use of my time. 
course e SEED courses have useful activities and content. 
(TF) e SEED courses increased my knowledge of best practice. 


SEED impact on 
teacher instruction 


What I learned from SEED taught me how to implement effective strategies in my 
classroom. 
SEED content is directly relevant to teaching and learning in my school. 
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Constructs 
(Data Sources) 


Items 


(TF) 


and student learning |e |use the knowledge gained from SEED in my work with students. 
(TF) e |am able to put into practice the ideas presented in SEED. 

e SEED has improved student learning in my classroom. 

e SEED has provided me with knowledge and skills to support high-needs students. 
Apply what has been | e On average, how often do you apply what you have learned in SEED in your 
learned from SEED in classrooms? 
classroom (TF) 
SEED utility to e SEED content is useful for my PD. 
support teacherPD |e The SEED PD is relevant to my needs. 
(TF) e The integration of the SEED PAK and RANDA is easy to use. 

e The integration of the SEED PAK and RANDA allowed me to better track my 

professional growth plans. 
e The integration of the SEED PAK and RANDA helped facilitate conversations with 
my principal about professional growth. 

Quality of C support | e My IC provided information and resources | needed to support my learning needs 
(TF) and professional goals. 

e My IC was available to answer questions. 

e My IC was knowledgeable about PD topics. 

e Overall, | was satisfied with the quality of the coaching and support | received. 
SEED impact on peer | e SEED has encouraged teachers to share what they learned with their colleagues. 
collaboration e Myschool provided opportunities for teachers to share information gained from 


SEED with colleagues. 

Teachers in my school took advantage of opportunities to share ideas, knowledge, 
and skills gained from SEED. 

SEED involved collaboration with teachers from other schools who teach at the same 
grade level and/or content area. 


Principal engagement 
in teacher PD 
(TF) 


My principals or supervisor recommended and planned PD based on needs 
identified through the teacher evaluation process. 

The current teacher evaluation systems in place provide information that allows my 
principal or supervisor to be supportive of teacher PD. 

My principal or supervisor understands what PD support teachers need. 

During conversations with my principal or supervisor regarding my professional 
growth (i.e., improvement on the teacher evaluation rubric and in knowledge and 
implementation of best practices), my principal is supportive. 


Table A13 shows Cronbach’s alphas for all teacher survey constructs over time. Overall, reliability 
estimates were high (a > 0.80) across all scales except the growth mindset scale. Specifically, the 
reliability estimate of the growth mindset scale was low and below the acceptable level (@ < 0.70). In 
this final report, data on growth mindset were excluded due to consistent low internal reliability of 


the measure. 


Appendix B-27 


Table A13. Reliability Estimates of Teacher Survey Constructs Over Time 


Pilot Full Implementation 


Baseline Baseline Y1 Y2 Y¥3 Y¥4 


Subscales (fall 2015) (spring (spring (spring (spring (spring 


rA)) PA) rAE)) PAE) 


Instructional practice- 
focused PD 

Data-driven PD 58 | 0.88 | 291 | 0.91 | 271 | 0.91 | 365 | 0.89 | 404 | 0.88 | 284 | 0.91 
School-provided time and 
resources to access PD@ 
Student learning-focused 


PD 

Growth mindset - | - - | ~ | 279 | 0.29 | 369 | 0.51 | 409 | 0.46 | 281 | 0.35 
Quality of SEED PAK® - | - - | ~ | - | -~ | 125 | 0.93 | 100 | 0.70 | 67 | 0.94 
Quality of SEED blog® - | - - | - - | - | 99 | 0.94 | 56 | 0.94 | 39 | 0.94 
Quality of SEED course? = = - = = = = SUE AU al uke) 
SEED impact on teacher 

instruction and student - - - - | 80 | 0.96 | 187 | 0.97 | 208 | 0.96 | 195 | 0.96 
learning 

SE uly do SHE -| - | = | = | 177] 092 | 239 | 0.93 | 159 | 0.88 | 138 | 830 
teacher PD 


Quality of IC support -- -- -- - 102 | 0.95 | 131 | 0.96 | 148 | 0.95 | 144 | 0.95 
SEED impact on peer 
collaboration 

a Sngagemieie -| - | - | = | = | = | 373 | 087 | 406 | 0.84 | 278 | 0.85 
eacher PD* 

a The construct includes two items. Hence, a is the correlation between these two items. 

> In fall 2017, a separate survey including three items assessing teacher perception of the quality of SEED PAK was sent to 
teachers to collect baseline data. 

¢New constructs added to the teacher follow-up survey in 2018. 

4A new construct added to the teacher follow-up survey in 2019. 

Note. Constructs measured by one item are not reported in this table. 


Student Engagement Survey 


Two different student engagement surveys were developed for students in 3" to 5” grades (SES:3-5) 
and 6" to 10 grades (SES:6-10) in May 2016. The same surveys were administered again with 
students in May 2017. For the 3“ to 5" grade survey, 17 items measuring four constructs of student 
engagement were included: (1) teacher-supported engagement in individual student learning (4 
items); (2) teacher-supported engagement in content learning (5 items); (3) positive attitude toward 
school (4 items), and (4) teacher support (4 items). Items measuring individual student learning and 
content learning were developed by McREL, and these items were rated on a 1 to 5 (strongly disagree 
to strongly agree) scale. Items measuring positive attitude toward school and teacher support were 
from the Seattle Social Development Project (Catalano et al., 2005; Hawkins & Catalano, 1990; 
Kusche & Greenberg, 1988). Response options were yes and no. 
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For the 6" to 10" grade survey, 22 items measuring five constructs of student engagement were 
included: (1) teacher-supported engagement in individual student learning (4 items); (2) teacher- 
supported engagement in content learning (5 items); (3) school engagement in math learning (4 
items); (4) school engagement in English learning (4 items); and (5) engagement in academic success 
(5 items). Items measuring individual student learning and content learning were developed by 
McREL and were rated on a 1 to 7 (completely disagree to completely agree) scale. Items measuring school 
engagement in math and English learning were from the Student Engagement Questionnaire 
(Dornbusch & Steinberg, 1990). Items measuring homework engagement were rated on a 1 to 6 
scale (none, about 15 minutes, about 3 minutes, about an hour, about 2 or 3 hours, about 4 hours). The 
remaining school engagement items were rated on a 1 to 5 scale (never, seldom, fairly often, usually, 
always). Items measuring engagement in academic success were adapted from (Landau, Oyserman, 
Keefer, & Smith, 2014), and these items were rated on a 1 to 7 (strongly disagree to strongly agree) scale. 


All students between 3“ and 10" grade from SEED schools were invited to take the anonymous 
student engagement survey online annually between spring 2016 and spring 2020. Each survey took 
about 10-15 minutes to complete. The overall response rates by grade level over time are presented 
in Table A14.*' The low response rate during the 2020 study survey may be attributed to the 
disruption caused by the 2020 COVID-19 pandemic. Also resulting from the pandemic, schools 
were closed in March and remained closed throughout the remainder of the school year. Therefore, 
in the 2020 survey, students were asked to give the rating based on their experience prior to the 
school closure. Due to the unique challenges during the COVID-19 when the 2020 student survey 
was administered, data collected in the 2020 school year may not be directly comparable to data 
collected in the prior years. 


Table Al 4. Student Survey Response Rate by Grade Level 


Response Rate 


Grade Level 2017 2018 2 
34 91.0% 82.8% 724% 75.6% 16.7% 
4th 93.7% 86.1% 79.6% 79.9% 33.2% 
5th 90.6% 85.5% 92.7% 85.5% 31.6% 
6th 87.2% 89.5% 70.9% 73.5% 37.2% 
7th 71.9% 82.7% 78.0% 63.6% 46.8% 
gin 39.6% 79.8% 71.8% 65.7% 40.1% 
gih 66.2% 56.0% 73.2% 64.2% 23.8% 
10th 52.5% 54.7% 62.5% 53.6% 27.6% 


Note. Data source for student enrollment number: httos://www.cde.state.co.us/cdereval/pupilcurrent. 

a Students (n = 6) who did not provide school information were removed from this analysis. 

b Students (n = 12) who did not provide school (n = 12) or grade (n = 1) information were removed from this analysis. 
¢ Students (n = 6) who did not provide grade information were removed from this analysis 


Table A15 shows the items by construct measured in SES:3-5 and SES:6-10. 


3! Data source for school enrollment number is school data from the Colorado Department of Education website 
(https: //www.cde.state.co.us/cdereval/pupilcurrentschool). 
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Table AI5. SES Survey Items by Construct by Grade Level 


Constructs Items 
SES:3-4 
Teacher-supported e My teachers help me do my best work. 
engagement: individual |e My teachers know how | like to learn. 
student learning e My teachers know how to help me with my work. 
e My teachers give me different kinds of work to help me learn. 
Teacher-supported e My teachers know a lot about what we are learning in class. 
engagement: content e My teachers help the class understand what we are learning. 
learning e My teachers know the answers to questions | have about what we are learning. 
e My teachers help me when | don’t know the answer. 
e _| learn a lot from my teachers. 
Positive attitude toward | e Do you think school is fun? 
school e Donice things happen to you at school? 
e Do you feel unhappy at school a lot? 
e Do you look forward to going to school? 
Teacher support e Does your teacher notice when you try hard? 
e Does your teacher tell you when you do well? 
e Does your teacher help you when you need it? 
e Dovyou like your teacher? 
Student engagement | ¢ Homework: In the <current school year> school year, how much time have you 
in math learning put into math homework each week?2 
e Attendance: In the <current school year> school year, how often do you skip/miss 
your math class (an unexcused absence)?>:° 
e Classroom Attention: In the <current school year> school year, how often do you 
really pay attention during your math class?° 
e Classroom Concentration: In the <current school year> school year, how often 
does your mind wander during your math class?°:° 
Student engagement |e Homework: In the <current school year> school year, how much time have you 
in English/language put into English/language arts homework each week, including reading and 
arts learning writing assignments?2 
e Attendance: In the <current school year> school year, how often do you skip/miss 
your English/language arts class (an unexcused absence)?> © 
e Classroom Attention: In the <current school year> school year, how often do you 
really pay attention during your English/lanquage arts class?° 
e Classroom Concentration: “In the <current school year> school year, how often 
does your mind wander during your English/language arts class?°.° 
Teacher-supported e My teachers help me do my best work. 
engagement: individual | « My teachers know how I like to learn. 
student learning e My teachers know how to help me with my work. 
e My teachers give me different kinds of work to help me learn. 
Teacher-supported e My teachers know a lot about what we are learning in class. 
engagement: content | My teachers help the class understand what we are learning. 
learning e My teachers know the answers to questions | have about what we are learning. 
e My teachers help me when | don’t know the answer. 
e | learn a lot from my teachers. 
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Constructs Items 


Engagement in e My academic success in the future begins now. 
academic success e have a hard time seeing myself as an academically successful student next 
year.° 
e My image of myself as an excellent student next year feels like a natural part of 
who | am now. 
e | feel connected to my image of myself as a successful student next year. 
e | caneasily see myself being an academically successful student next year. 


a Each item was rated on a 5-point Likert-type scale; the responses ranging from strongly disagree (1) to strongly agree (5). 
> Each item was rated yes (2) or no (1). 
¢ The item was reverse coded. A higher score means better outcome. 


Table A16 shows Cronbach’s alphas for all subscales throughout the full implementation period. 
The scales with a Cronbach’s alpha lower than 0.70 are highlighted in orange in Table A16. The 
constructs that have acceptable internal consistency (a > = 0.70) between baseline and Y3 are (1) 
teacher-supported engagement in content learning for both 3-5" and 6"—10" graders; (2) teacher- 
supported engagement in individual student learning for 6"—10" graders; and (3) engagement in 
academic success for 6-10" graders. 


Table A16. Reliability Estimates of SES:3-5 and SES:6-10 Subscales 


Full Implementation 


Subscales ae ae ee 
(spring 2016) (spring 2017) | (spring 2018) (spring 2019) (spring 2020) 


n | n a n a 


SES:3-5 
Teacher-supported 
engagement in individual 1433 
student learning 
Teacher-supported 
engagement in content 1409 
learning 

Positive attitude toward 
school? 
Teacher support 


1322 | 0.75 | 419 | 0.80 


1456 


Teacher-supported 
engagement in individual 1687 | 0.84 | 1899 | 0.84 | 2025 | 0.86 | 1889 
student learning 
Teacher-supported 
engagement in content 1661 | 0.91 | 1869 | 0.91 | 2019 | 0.92 | 1880 
learning 

Student engagement in 
math? 

Student engagement in 
English> 

Engagement in academic 
siceeee 1653 | 0.82 | 1862 | 0.83 | 2017 | 0.84 | 1864 


a One item was worded negatively. The item was reverse-coded when calculating the scale reliability. 


1703 1916 2010 1871 


1671 1884 1995 1876 


Appendix B-31 


Full Implementation 
Baseline Y1 Y2 Y3 Y4 
(spring 2016) | (spring 2017) | (spring 2018) | (spring 2019) — (spring 2020) 
n a n qa (7 a n a n a 
> All items were measured on a six-point scale, except that one item was measured on a seven-point scale. The item was 
rescaled to a six-point scale when calculating the reliability. Also, two items were worded negatively, and these two items 
were reverse-coded when calculating the scale reliability. 


Subscales 


Project Record and Extant Data 


Several project records and extant data were included in the evaluation, including 


e IC syllabi and TLC materials 

e Project staff self-reflection of implementation progress and challenges 

e PD committee and content development meeting minutes” 

e Teacher participation in SEED data (Le., IC participation logs) 

e Teacher participation in Content Development Teams data 

e PD completion data 

e SEED PAK resources 

e Data related to linkage of SEED platform and COPMS (RANDA) 

e TLC participation gradebook data 

e COPMS data (teacher evaluation data) 

e Individual-level student demographic data and achievement data in math and ELA for 
students from the participating schools 

e School-level demographic data and student achievement data in math and ELA for students 
from comparison schools 


e Other project implementation records maintained by the project team 


These data were provided by the project staff, except student achievement data” that were provided 
by the Colorado Department of Education (CDE) and teacher COPMS data that were provided by 
RANDA Solutions. Publicly available data released by CDE were also used in the evaluation as 
appropriate. For instance, the following CDE websites were frequently used to access school 
demographic and school-level student achievement data: 


e https://www.cde.state.co.us/cdereval/staffcurrent 


e = http://www.cde.state.co.us/cdereval/rvprioryearpmdata 
e = https://www.cde.state.co.us/cdereval/tvprioryearhrdata 
e = https://www.cde.state.co.us/cdereval/pupilcurrent 


e = http://www.cde.state.co.us /code/accountability-dataexplorertool 


32 These were formal meeting minutes that were taken to document the agenda items, decisions made around them, and 
any other action items. 

33 A data-sharing agreement (DSA) between McREL and CDE was established to access student achievement data for 
the evaluation. The DSA allows McREL to access student-level achievement data for all SEED schools and school-level 
achievement data for the selected matched comparison schools. 
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Due to the 2020 COVID-19 pandemic, student achievement data and teacher evaluation data were 
not available for the 2019-20 school year. 


Data Analysis Plan 


All evaluation data collected were reviewed and cleaned prior to any analyses. Data cleaning 
consisted of reviewing all project records and surveys and identifying information gaps, 
discrepancies, and redundancies. Instances in need of clarification or further cleaning were managed 
on a case-by-case basis. Before quantitative data analyses were performed, evaluators screened the 
data for data entry errors and improbable responses. Descriptive statistics (e.g., frequencies, 
percentages, means, standard deviations, or cross-tabulations) were examined for all quantitative 
data collected, including principal, teacher, student surveys as well as COPMS (RANDA) data and 
student achievement data. 


This section is organized into four sub-sections. The first discusses the process to determine 
baseline data for different survey data sources. The second discusses the approach to convert 
reading and math scale scores to z-scores. The third presents the data analysis plan and strategies for 
each evaluation question. The last section describes the method to calculate and report on effect size 


(ES). 
Baseline Data 


Because the project design involves three phases—development (spring 2015), pilot (2015-16), and 
full implementation (2016-17 through 2019—20)—baseline data were collected on different timelines 
depending on the outcomes of interest and data sources. Overall, for the principal survey, data 
collected during the pilot (2015-16) were used as baseline since all principals and assistant principals 
started participating in SEED at the start of the pilot. For the student engagement survey and 
student achievement data, data collected in spring 2016 were used as baseline since the full 
implementation started in 2016-17 school year. The baseline timeline for the teacher survey data 
was more complicated because only a subgroup of teachers participated in SEED during the pilot. 
Additionally, as the project continued to progress and mature, additional data of interest were 
collected after the pilot (see Data Collection Methods section for details). 


Preliminary analyses were conducted to determine the baseline data for the teacher survey. Pilot 
teachers’ and non-pilot teachers’ baseline data were collected at different time points (pilot teachers’ 
baseline data were collected in fall 2015 while non-pilot teachers’ baseline data were collected in 
spring/summer 2016). Group differences between pilot teachers’ and non-pilot teachers’ 
perceptions and experiences at baseline were examined. Specifically, a series of multilevel models 
was estimated to account for the nested nature of the datasets (i.e., teachers nested within schools). 
Results indicated that pilot teachers and non-pilot teachers did not differ in the following constructs 
at baseline (Le., before exposure to SEED): 


e Instructional practice-focused PD (6 = 0.26, SE = 0.14, p = 0.061) 

e Data-driven PD (6 = -0.80, SE = 0.15, p = 0.590) 

e School-provided time and resources to access PD (6 = -0.04, SE = 0.17, p = 0.984) 
e Student learning-focused PD (6 = 0.09, SE = 0.13, p = 0.516) 

e Opportunity to collaborate (6 = 0.11, SE = 0.16, p = 0.474) 

e Up-to-date PD (6 = 0.18, SE = 0.218, p = 0.218) 

e Evidence-based PD (6 = 0.08, SE = 0.15, p = 0.579) 
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These findings suggest that it is appropriate to combine pilot teachers’ and non-pilot teachers’ 
baseline data. Analyses were conducted to examine if teachers’ perceptions on these constructs 
changed over time. A series of multilevel models (i.e., repeated measures nested within individuals, 
and individual nested within schools) was conducted to account for the nested data structure. 
Hedge’s g was calculated to show the magnitude of change between baseline and the end of Year 1 
full implementation. 


In terms of the follow-up measures, pilot teachers’ experiences with SEED were first assessed in 
2016 after one year of SEED participation, and the same data were collected again in 2017 after two 
years of SEED participation. For non-pilot teachers, their experiences with SEED were first 
examined in 2017 after one year of SEED participation. A total of 27 pilot teachers responded to 
both 2016 and 2017 surveys; paired-sample t-tests were conducted to examine changes in these pilot 
teachers’ experiences with SEED. Results showed that these pilot teachers’ experience with SEED 
did not change in the following constructs from 2016 to 2017. 


e SEED impact on teacher instruction and student learning (f= 1.96, df= 10, p = 0.079) 
e Apply what has been learned from SEED in classroom (¢= 0.80, df= 11, p = 0.438) 

e SEED content utility and PD relevance (¢ = 2.15, df= 11, p = 0.054) 

e SEED PAK and RANDA utility to support teacher PD (¢= 1.09, df= 18, p = 0.290) 

© Quality of IC support (¢ = 1.00, df= 13, p = 0.336) 

e SEED impact on peer collaboration practices (¢= 0.99, df= 22, p = 0.332) 


A series of multilevel models was also conducted to examine the differences between pilot teachers 
and non-pilot teachers regarding their SEED experiences measured in 2017. Results showed that 
pilot and non-pilot teachers’ experiences with SEED did not significantly differ in the following 
constructs. 


e SEED impact on teacher instruction and student learning (6 = -0.08, SE = 0.23, p = 
0.721) 

e Apply what has been learned from SEED in classroom (6 = -0.23, SE = 0.26, p = 0.382) 

e SEED content utility and PD relevance (6 = -0.05, SE = 0.23, p = 0.825) 

e SEED PAK and RANDA utility to support teacher PD (6 = 0.11, SE = 0.27, p = 0.693) 

¢ Quality of IC support (6 = 0.17, SE = 0.20, p = 0.386) 

e SEED impact on peer collaboration practices (@ = 0.22 SE = 0.23, p = 0.350) 


It should be noted that pilot teachers were only exposed to a semester-long TLC during the pilot, so 
it is not totally unexpected to find that their responses about theit experiences with SEED in spring 
2017 were not significantly different from those who started participating in SEED in the 2016-17 
school year. Hence, the spring 2017 data collected from all teachers, including pilot teachers, were 
analyzed and reported as the Y1 follow-up data. 
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Converting Scale Scores to Z-scores 


Student achievement data were obtained from the CDE website. Between 2016 (baseline) and 2019 
(Y3 outcome), student achievement data that were available from CDE and used in the analyses 
include:* 


e Partnership for Assessment of Readiness for College and Careers (PARCC) ELA 
and math tests scores for students from third to eighth grade for the 2015-16 and 
2016-17 school years 

e Colorado Measures of Academic Success (CMAS) ELA and math tests scores for 
students from third to eighth grade for the 2017-18 and 2018-19 school years 


e Preliminary SAT (PSAT) ELA and math test scores for students in 10" grade 


The CDE website provides within-grade school mean scores for all students as well as student 
subgroups. To allow for cross-grade level comparisons, the within-grade school mean scores were 
converted to z-scores using the state means and standard deviations calculated from individual 
student-level data within grade within the test year. The formula for converting scale scores to z- 
scores is 


Zio, = (Vic) - StateMeanStudentsc) / StateS DS tudentsic 
where 


Zi = the z-score from the i™ time point 
in the G® grade in the ile school 


Yi; = the observed reading school-mean 
reading score from the i" time 
point in the G™ grade in the j® 


school; 


StateMeanS tudentsic = the population mean of reading scores of all 
students in the state at the i time point in the ee 
grade 


StateS DStudentsic = the population standard deviation of reading 
scores of all students in the state at the i® time 
point in the G™ grade. 


After z-scores were calculated, a school-level mean ELA score and a school-level mean math score 
were calculated for each school. Student achievement data collected during the 2015-16 school year 
were used as the baseline data for the matching. The 2016-17, 2017-18, and 2018-19 data were 


+4 Tn 2018, CDE transitioned the state standardized tests from PARCC to CMAS. Therefore, the baseline data (2015-16) 
and the Y1 student achievement outcome (2016-17) for the impact analyses were measured by PARCC. Starting with 
the 2017-18 school year (Y2 outcome), student outcomes were measured by CMAS. According to the technical report 
released by CDE, the correlations between PARCC scales and CMAS scales were greater than 0.98 in math and greater 
than 0.97 in ELA. The high correlations between the baseline tests and the Y2—3 outcome measures meet the minimal 
requirement specified by the WWC standards—correlation between the baseline measurement and outcome 
measurements should be 0.60 or higher. Additionally, starting in the 2017—18 school year, CMAS was no longer 
administered to 9'-grade students in the State of Colorado. To allow for longitudinal comparison, this report only 
included the CMAS data collected from 3" to 8'-grade students during the project period. 
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used as the Y1, Y2, and Y3 outcome measures, respectively. The same procedure was applied to all 
six independent QED studies with corresponding data from student groups. 


Data Analysis Strategies by Evaluation Questions 


Data analysis strategies used to answer each evaluation question ate summarized in Table A17. The 
outcomes of interest for each question are also specified in the table. Items used to measure the 
constructs (outcomes of interest) ate provided in Tables A9 (principal-reported outcome measures) 
A12 (teacher-reported outcome measures), and A13 (student-reported outcome measutes). 


> 


Table Al7. Outcomes of Interest and Data Analysis Strategies by Evaluation 


Question 


Evaluation Question | 


Outcomes of Interest 


Data Analysis Plan 


Goal 1. Successfully implement and continually improve SEED in up to 30 NW BOCES schools during a 


four-year period (timeliness and continuous improvement) 


a. Are project activities occurring as 
planned? 

b. Is the project maintaining its 
planned rigor? 

c. What barriers and unanticipated 
outcomes were encountered, and 
what revisions were made during 
implementation? 

d. What were key project successes 
during implementation? 


a. Are principals more engaged in 
teacher professional growth? 


Fidelity of implementation; 
implementation challenges; 
implementation successes 


Principal engagement in 
teacher PD 


b. Are principals able to provide 
more and better support to 
teachers through SEED? 


SEED utility to support teacher 
PD 


Project record review and content 
analysis; descriptive analysis of SEED 
participation data. Data collected 
throughout the project period were 
included in the analyses. 


Goal 2. Improve principal engagement in teacher professional growth and support. 


Latent growth curve modeling 
(LGC). Individual responses (i.e., 
principal and teacher responses) were 
aggregated to the school level and 
used in the analysis. LGC was 
conducted to examine the rate of 
change over time. 


c. Do SEED teachers, in 
comparison with non-SEED 
teachers, report a higher level of 
principal engagement in teacher 
professional growth?* 


Principal engagement in 
teacher PD 


Hierarchical linear modeling (HLM). 
Individual responses (i.e., teacher 
responses) were used in the analysis. 
Multilevel modeling was conducted to 
examine the difference between SEED 
and non-SEED teachers while 
accounting for the nested structure of 
the data. Data collected during the 
2018-19 school year were used in the 
analyses. Regardless of the SEED 
participation status in the 2018-19 
school year, teachers might have prior 
exposure to SEED before then. 
Therefore, teachers’ prior participation 
data during the 2016-17 and 2017-18 
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Evaluation Question 


Outcomes of Interest 


DELEWAUE Sma CU 
school years were used to determine 
their prior experience with SEED.*5 


d. What is the relationship between 
school-level SEED participation 
rates and principal and teacher 
reports of principal engagement 
in teacher PD?* 


Principal engagement in 
teacher PD 


Regression analysis. Level of SEED 
implementation was measured by the 
percentage of teachers who were 
defined as SEED participants within 
each school. Data collected from 
2018-19 were used for the analysis 
(Y3 full implementation). A series of 
linear regression analyses was 
conducted to understand the 
associations between levels of SEED 
participation and the outcomes of 


e efele 
a. Are teachers utilizing SEED? 
Which activities are teachers 
using? Does use increase over 
time? 


SEED participation in various 
SEED activities 


interest.26 


Descriptive analysis of SEED 
utilization, SEED participation, 
COPMS and IC participation data. 
Data collected throughout the project 
period were used in the analyses. 


b. Is PD reflective of best practices 
and up-to-date content 
knowledge? 


e Up-to-date PD 

e Evidence-based PD 

e School offering evidence- 
based PD 


c. Do teachers have time and 
resources to access PD? 


e School-offered time and 
resources to access PD 


d. Is PD offered to teachers driven 
by data and does it meet the 
needs of individual teachers? 


e Data-driven PD 
e School-offered 
individualized teacher PD 


LGC. Individual responses (i.e., 
principal and teacher responses) were 
aggregated to the school level and 
used in the analysis. LGC was 
conducted to examine the rate of 
change over time. 


e. Do SEED teachers, in 
comparison with non-SEED 
teachers, give higher ratings 
regarding the extent to which (1) 
PD is reflective of best practice 
and up-to date content 
knowledge; (2) PD offered is 
data-driven and aligned with 
individual needs; and (3) 
teachers have time and 
resources to access PD?* 


Up-to-date PD 
Evidence-based PD 
Data-driven PD 
School-offered 
individualized teacher PD 
e School-offered time and 
resources to access PD 


HLM. Individual responses (i.e., 
teacher responses) were used in the 
analysis. Multilevel modeling was 
conducted to examine the difference 
between SEED and non-SEED 
teachers while accounting for the 
nested structure of the data. Data 
collected during the 2018-19 school 
year were used in the analyses. 
Regardless of the SEED participation 
status in the 2018-19 school year, 
teachers might have prior exposure to 


35 A variable indicating teachers’ prior experience with SEED was created and entered in the final analytic model as a 
covariate (0 = never participated in SEED; 1 = had 1 year of experience with SEED prior to 2018-19; 2 = had 2 years 
of experience with SEED prior to 2018-19). 
*° Baseline data were entered in the regression models as a covariate. Specifically, 2016 data were used as baseline data 
for the teacher survey, teacher evaluation data, and student survey data; 2015 data were used as baseline for principal 


sutvey data. 
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Evaluation Question 


Outcomes of Interest 


PEL EWAUE I SLcaa CU 
SEED before then. Therefore, 
teachers’ prior participation data 
during the 2016-17 and 2017-18 
school years were used to determine 
their prior experience with SEED.*” 


f. What is the relationship between 
school-level SEED participation 
rates and teachers’ ratings on the 
extent to which (1) PD is 
reflective of best practice and up- 
to date content knowledge; (2) 
PD offered is data-driven and 
aligned with individual needs; and 
(3) teachers have time and 
resources to access PD?* 


a. Does teachers’ knowledge of 


Up-to-date PD 
Evidence-based PD 
Data-driven PD 
School-offered 
individualized teacher PD 
e School-offered time and 
resources to access PD 


e Instructional practice- 


practice change in the 
classroom? 


best practices increase? focused PD 
e Student learning-focused 
PD 
b. Do teachers demonstrate SEED teachers only: 


e SEED impact on teacher 
instruction and student 
learning 

e Teachers applied what 
they learned from SEED in 
classroom 

e SEED impact on peer 
collaboration 


All teachers: 


e Opportunity to collaborate 
e Teacher evaluation ratings 


Regression analysis. Level of SEED 
implementation was measured by the 
percentage of teachers who were 
defined as SEED participants within 
each school. Data collected from 
2018-19 were used for the analysis 
(Y3 full implementation). A series of 
linear regression analyses was 
conducted to understand the 
associations between levels of SEED 
participation and the outcomes of 
interest.%8 


Goal 4. Support teachers in successfully implementing SEED practices and content. 


LGC. Individual responses (i.e., 
teacher responses) were aggregated 
to the school level and used in the 
analysis. LGC was conducted to 
examine the rate of change over time. 


c. DoSEED teachers, in 
comparison with non-SEED 
teachers, (1) give higher ratings 
on their knowledge of best 
practices, (2) have more 
opportunity to collaborate with 


e Instructional practice- 
focused PD 

e Student learning-focused 
PD 

e Opportunity to collaborate 

e Teacher evaluation ratings 


HLM. Individual responses (i.e., 
teacher responses) were used in the 
analysis. Multilevel modeling was 
conducted to examine the difference 
between SEED and non-SEED 
teachers while accounting for the 


37 A variable indicating teachers’ prior experience with SEED was created and entered in the final analytic model as a 
covariate (0 = never participated in SEED; 1 = had 1 year of experience with SEED prior to 2018-19; 2 = had 2 years 
of experience with SEED prior to 2018-19). 
** Baseline data were entered in the regression models as a covariate. Specifically, 2016 data were used as baseline data 
for the teacher survey, teacher evaluation data, and student survey data; 2015 data were used as baseline for principal 


sutvey data. 
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Evaluation Question 
job-alike peers, and (3) have 
higher teacher evaluation 
scores?* 


Outcomes of Interest 


PEL EWAUE I SLma CU 
nested structure of the data. Data 
collected during the 2018-19 school 
year were used in the analyses. 
Regardless of the SEED participation 
status in the 2018-19 school year, 
teachers might have prior exposure to 
SEED before then. Therefore, 
teachers’ prior participation data 
during the 2016-17 and 2017-18 
school years were used to determine 
their prior experience with SEED.*9 


d. What is the relationship between 
school-level SEED participation 
rates and (1) teachers’ ratings on 
their knowledge of best practices, 
(2) teachers’ opportunity to 
collaborate with job-alike peers, 
and (3) teacher evaluation 
scores?* 


a. To what extent do SEED schools 
and comparison schools differ in 
student achievement outcomes 
after one year, two years, and 
three years of full 
implementation? 


Instructional practice- 
focused PD 

Student learning-focused 
PD 

Opportunity to collaborate 
Teacher evaluation ratings 


School-level reading 
outcome 
School-level math 
outcome 


Regression analysis. Level of SEED 
implementation was measured by the 
percentage of teachers who were 
defined as SEED participants within 
each school. Data collected from 
2018-19 were used for the analysis 
(Y3 full implementation). A series of 
linear regression analyses was 
conducted to understand the 
associations between levels of SEED 
participation and the outcomes of 
interest. 


Goal 5. Improve student achievement. 


Regression analysis. A series of 
linear regression analyses was 
conducted to understand the 
difference between SEED and non- 
SEED schools at the end of Year 1, 
Year 2, and Year 3. All covariates 
used in the matching process were 
included in the final analytical model 
(see the impact analytic model 
following this table). Data collected 
between the 2015-16 and 2018-19 
school years were included in the 
analysis. See the impact analytic 
model following this table. 


b. To what extent do students from 
SEED schools demonstrate an 
increase in student achievement 
outcomes over time?* 


School-level reading 
outcome 
School-level math 
outcome 


LGC. School-level student 
achievement data were used in the 
analysis. Conditional LGC was 
conducted to examine the rate of 


39 A variable indicating teachers' prior experience with SEED was created and entered in the final analytic model as a 


covariate (0 = never participated in SEED; 1 = had 1 year of experience with SEED prior to 2018-19; 2 = had 2 years of 
experience with SEED prior to 2018-19). 


40 ‘ ' ‘ : : : 
Baseline data were entered in the regression models as a covariate. Specifically, 2016 data were used as baseline data 
for the teacher survey, teacher evaluation data, and student survey data; 2015 data were used as baseline for principal 


sutvey data. 
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Evaluation Question 


Outcomes of Interest 


PELE WAUE IN CLaa CU 
change over time while controlling for 
other school-level covariates. 

Data collected between the 2015-16 
and 2018-19 school years were 
included in the analysis. 


To what extent do high-needs 
students from SEED schools and 
comparison schools differ in 
student achievement outcomes 
after one year, two years, and 
three years of full 
implementation? 


School-level reading 
outcome of high-needs 
students 

School-level math 
outcome of high-needs 
students 


Regression analysis. A series of 
linear regression analyses was 
conducted to understand the 
difference between high-needs 
students within SEED and those within 
non-SEED schools at the end of Year 
1, Year 2, and Year 3. High-needs 
students include students from 
racial/ethnic minority backgrounds and 
students receiving free or reduced 
lunch. All covariates used in the 
matching process were included in the 
final analytical model. Data collected 
between 2015-16 and 2018-19 school 
years were included in the analysis. 
See the impact analytic model 
following this table. 


To what extent do high-needs 
students from SEED schools 
demonstrate an increase in 
student achievement outcomes 
over time?* 


School-level reading 
outcome of high-needs 
students 

School-level math 
outcome of high-needs 
students 


LGC. School-level high-needs student 
achievement data were used in the 
analysis. Conditional LGC was 
conducted to examine the rate of 
change over time while controlling for 
school-level covariates. High-needs 
students include students from 
racial/ethnic minority backgrounds and 
students receiving free or reduced- 
price lunch. Data collected between 
the 2015-16 and 2018-19 school 
years were included in the analysis. 


Do schools with higher levels of 
teacher participation have higher 
student achievement?* 


School-level reading 
outcome 
School-level math 
outcome 


Regression analysis. Level of SEED 
implementation was measured by the 
percentage of teachers who were 
defined as SEED participants within 
each school. Data collected from 
2018-19 were used for the analysis 
(Y3 full implementation). A series of 
linear regression analyses was 
conducted to understand the 
associations between levels of SEED 
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Evaluation Question Outcomes of Interest Data Analysis Plan 


participation and the outcomes of 
interest.4" 

a. Does student engagement e For G3-5: Teacher- LGC. Individual responses (i.e., 
increase among participating supported engagement: principal and teacher responses) were 
schools? individual student learning; | aggregated to the school level and 

teacher-supported used in the analysis. LGC was 
engagement: content conducted to examine the rate of 
learning; positive attitude | change over time. Data collected 
toward school; and between the 2016-17 and 2018-19 
teacher support were included in the analysis.*2 

b. Do schools with higher levels of e For G6-10: Student Regression analysis. Level of SEED 
teacher participation have higher engagement in math implementation was measured by the 
levels of student engagement?* learning; student percentage of teachers who were 

engagement in ELA defined as SEED participants within 
learning; teacher- each school. Data collected from 
supported engagement: 2018-19 were used for the analysis 
individual student learning; | (Y3 full implementation). A series of 
teacher-supported linear regression analyses was 
engagement: content conducted to understand the 
learning; and engagement | associations between levels of SEED 
in academic success participation and the outcomes of 
interest. 


Note. Due to the 2020 COVID-19 pandemic, student achievement and teacher evaluation data were not available for the 
2019-20 school year. For the same reason, during the 2019-20 school year, the survey response rates from principals, 
teachers, and students were much lower than the response rates in the prior years (see Tables A8, A11, and A14). Therefore, 
data collected during the 2019-20 school year were excluded from the analysis. 


SEED Participants. Within SEED Schools, teachers were able to participate in SEED in various 
ways, from taking formal SEED courses to reading a SEED blog or complete a SEED PAK 


resoutce. Of various types of SEED activities, certain types of activities were more likely to make an 
impact on teacher practices and outcomes, including 


e SEED course participant (individual who participated in at least 80% of the TLC, PPS, 
independent study, and book study) 

e SEED PAK participant (individual who completed at least eight SEED PAK resources 
within a school year) 

e SEED actor (individual who reported applying what he/she has learned from SEED in 
the classroom on a weekly or daily basis) 

e SEED facilitator (individual who served as a SEED facilitator) 


When comparing the difference between SEED teachers and non-SEED teachers (i.e., evaluation 
questions that compare the difference between SEED and non-SEED teachers), teachers who met 


“! Baseline data were entered in the regression models as a covariate. Specifically, 2016 data were used as baseline data 
for the teacher survey, teacher evaluation data, and student survey data; 2015 data were used as baseline for principal 
sutvey data. 

* Due to the interruption caused by the COVID-19 pandemic, the survey response rate was low in the 2019-20 school 
year. Therefore, the 2019-20 student engagement data were excluded from the analysis. 
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at least one of the above criteria were identified as SEED teachers. Teachers who did not meet any 
of the above criteria were identified as non-SEED teachers. 


Level of SEED Participation. School-level SEED participation rates were calculated based on 
teacher participation data collected in Y3 (2018-19). Schools with 50% or more of teachers who 
participated in SEED (as defined above) were coded as high participation (HP) schools, and schools 
with less than 50% of teachers participated in SEED were coded as low participation (LP) schools. 
For this part of the analyses, teachers who participated in SEED during the 2018-19 school year 
were included in the analysis. In 2019, the overall teacher participation rate across SEED schools 
ranged from 15% to 92%. Across all SEED schools (” = 20), eight (40%) were HP schools. A series 
of regression analyses was conducted to examine if principal perception of teacher PD, teacher 
perception of PD, teacher evaluation scores, and student engagement differ between HP and LP 
schools. 


Analytic Models of Impact Studies. Six impact studies were conducted to examine the impact of 
SEED on student achievement, including three groups of students (all students within SEED 
schools, minority students, and FRL students) on two subject areas (reading and math). For each 
impact study, six sub-questions were examined: 


e To what extent do SEED schools and comparison schools differ in reading outcomes after 
one year of full implementation? 

e To what extent do SEED schools and comparison schools differ in reading outcomes after 
two years of full implementation? 


e To what extent do SEED schools and comparison schools differ in reading outcomes after 
three years of full implementation? 

e To what extent do SEED schools and comparison schools differ in math outcomes after 
one year of full implementation? 


e To what extent do SEED schools and comparison schools differ in math outcomes after 
two years of full implementation? 

e To what extent do SEED schools and comparison schools differ in math outcomes after 
three years of full implementation? 


Within each impact study sample, a series of linear regression models was conducted. School-level 
student achievement data were used in the analyses; hence, the regression model was a one-level 
model. When conducing the analyses, all covariates used in the matching were entered in the first 
block, and treatment status (i.e, SEED variable) was entered in the second block. R’ and changes in 
R’ after entering the treatment status in the model were both reported. The simplified analytical 
model for the impact analyses is presented below. 


Y = Bo + fi-« (Covariates) + B11 (SHEED) + 


Where 
ei = the standardized mean score in math or language arts 
Covariates. = the baseline school-level covariates used in 
the matching 
SEED — =school treatment status (1 = SEED, 0 = comparison) 


‘43 The list of covariates used in the matching are described in the Study Samples section (see Tables A4 and A5). 
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Bice = the effect of school level covariates 
Beat = the treatment effect 


e = the error term 


Effect Size Calculation 


This report follows What Works Clearinghouse reporting guidelines when calculating effect sizes for 
impact studies (What Works Clearinghouse, 2017). Specifically, because the outcomes wete 
converted to z-scores, the mean difference between the groups is the effect size. According to Wolf 
(1986), an ES value of 0.25 or greater is educationally significant, and an ES value of 0.50 is of 
practical significance. 
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Appendix B. Assessment of Fidelity of Implementation 


Table B!I. SEED Fidelity of Implementation Matrix 
Note. Unless otherwise noted, when a score was assigned based on percent: 1= low (0-33%); 2 = med (34-66%); 3 = high (67—100%). 


Indicators 


Definition 


Unit of 
implementation 


Data 
Source(s) 


Data 
Collection 
(who, when) 


Score for levels 
(0) 
implementation 
at unit level 


Xo LUT oh ce) 
next higher 
level if 
needed 
(score and 
threshold): 
Indicate 
level 


Threshold for 
adequate 


implementation 
at unit level 


Roll-up to 
program level 
(score and 
threshold for 
adequate 
implementation 
at sample level) 


Expected 
sample for 
fidelity 
measure 


Expected 
years of 
fidelity 
measurement 


1. Topics for Learning Program PD committee | By June, Percent of 4 NIA Percent of Program Years 1-4 3 
learning strands and content meeting learning strands learning strands (2016-2020) T1 | (3/3 = 
strands for chosen development | minutes put that meet criteria. that meet criteria. 100%) 
PD are based on team(CDT) —_|_on Google Score based on Score based on 3 
selected teacher meeting Drive percent: (1) low; percent: (1) low; T2 | (2/2 = 
based on need minutes (2) med; (3) high. (2) med; (3) high. 100%) 
teacher Threshold=3. 3 
needs T3 | (2/2 = 

100%) 
3 

T4 | (2/2= 
100%) 

2. Evidence- PD activity | PD activity SEED PAK By Sept., Score assigned to | 1 % of 20 % of 20 randomly | SEED PAK Years 1-4 3 
basedPDis_ | has resources McREL each activity: randomly selected SEED resources (2016-2020) a (16/20 
selected by theoretical reviewed to reviews PD | 9 = no indication selected PAK resources throughout = 
ICs and evidence determine against of theoretical SEED PAK with score of 1. the project 80%) 
CDTs for base whether it research- base: and 1 = resources Score based on 3 
inclusion in behind it contains based theory dentine’ with score of | percent: (1) low; 12 (20/20 
the SEED strategies that theoretical base 1. Score (2) med; (3) high. e 
PAK are supported based on Threshold = 3 100%) 

by research- percent: (1) 3 
based theory low; (2) med; 13 (18/20 
(3) high. E 
Threshold = 90%) 
3 3 
TA ! 6/20 
80%) 

3.1Cs and TLCs Program ICs’ syllabi for | By Sept.and | Percentof TLCs | 3 N/A Percent of TLCs Program Years 1-4 3 
CDTs create | include at the TLCs Feb., SEED | that include at that include at (2016-2020) T1 | (6/6 = 
in-person least two in- TLC syllabus | |east 2 in-person least 2 in-person 100%) 
TLC activities | person activities per activities per T2|3 
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activities put on semester. Score semester. Score (5/5 = 
per Google Drive | based on based on 100%) 
semester percent: (1) low; percent: (1) low; 3 
(2) med; (3) high. (2) med; (3) high. T3 | (3/3 = 
Threshold = 3 100%) 
3 
T4 | (3/3 = 
100%) 
4.1Cs and TLCs that Program ICs’ syllabi for | By Sept.and | Percent of TLCs N/A Percent of TLCs | TLCs Years 1-4 3 
CDTs create have at the TLCs Feb., TLC that have a least that have a least (2016-2020) 11 (6/6 = 
virtual TLC least one syllabus one hour per one hour per 100%) 
activities hour per posted on week of virtual week of virtual 3 
week of Google Drive | activities. Score activities. Score T2 | (5/5 = 
virtual based on based on 100%) 
activities percent: (1) low; percent: (1) low; 3 
(2) med; (3) high. (2) med; (3) high. T3 | (3/3 = 
Threshold = 3 100%) 
3 
T4 | (3/3 = 
100%) 
5. ICs format TLCs utilize | Program TLC reviewed | Feb and Percent of TLCs NIA Percent of TLCs Program Years 1-4 3 
TLC delivery | the four against Sep., McREL | that utilize the that utilize the (2016-2020) T1 | (6/6 = 
using evidence- Vescio et al. reviews each | four evidence- four evidence- 100%) 
evidence- based TLC (2008) TLC TLC’s based TLC based TLC 3 
based standards standards syllabus put standards. Score standards. Score T2 | (5/5 = 
delivery from Vescio on the based on based on 100%) 
standards et al. (2008) Google drive percent: (1) low; percent: (1) low; 3 
model. (2) med; (3) high. (2) med; (3) high. T3 | (3/3 = 
Threshold = 3. 100%) 
3 
T4 | (3/3 = 
100%) 
Total Score Mean of the five T1 | 3 
for SEED PD indicators. T2 | 3 
Creation Threshold=3. T3 | 3 
Component T4 | 3 
6.ICs facilitate | TLC is Program ICs syllabi for | By Aug. and | Percent of N/A Percent of Program Years 1-4 3 
blended hosted for TLCs and Jan., learning strands learning strands (2016-2020) T1 | (6/6 = 
learning each content documents that have TLCs that have TLCs 100%) 
TLCs learning development | are put on hosted as hosted as 3 
strand team meeting | Google Drive | indicated in indicated in T2 | (5/5 = 
minutes syllabi. Score syllabi. Score 100%) 
determining based on based on 3 
learning percent: (1) low; percent: (1) low; T3 | (3/3 = 
strands (2) med; (3) high. (2) med; (3) high. 100%) 
Threshold = 3. 3 
T4 | (3/3 = 
100%) 
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Total Score Same as T1 | 3 
for SEED PD indicator 6. T2 | 3 
Delivery Threshold=3. T3 | 3 
Component T4 | 3 
7. SEED PDis | Principal or | Teachers SEED system | By June, 0=No School level: | Percent of All schools in | Years 1-4 1 
linked to teacher linking SEED 1 = Yes % of schools that have | which the (2016-2020) 1 (0/21 
specific assigns at COPMS with | COPMS data teachers score of 2. Score | intervention = 
teacher least one SEED will be with score of | based on is being 0%)*5 
performance | SEED PD platform provided to 1 in each percent: (1) low; implemented 1 
indicators by | in COPMS McREL school. (2) med; (3) high. 12 (7/21 
teachers or Score based | Threshold = 3. 7 
principals on percent: 33%)* 
(1) low (0- 2 
19%); (2) 3 (7/20 
med (20- = 
50%): (3) 35%)*” 
high (51- 1 
100%). (2/20 
Threshold = 4). 
2 10%)48 
8. Teachers Teacher Teachers SEED By June, 0=No School level: | Percent of All schools in | Years 1-4 flee 
complete completes platform PD SEED 4 = Yes % of schools that have | which the (2016-2020) aT (6/21 
SEED PD?9 at least one completion platform data teachers score of 2. Score intervention = 
facilitated data and will be with score of | based on is being 29%) 
SEED PD Innovation provided to 1 in each percent: (1) low; | implemented 23 
OR self- Coach TLC McREL and school. (2) med; (3) high. q2 | (9/21 
directed gradebook by January Score based | Threshold = 3. = 
completion data and 43%) 


score of 2. 


score of 2. 


score of 2. 


score of 2. 


45 Within SEED schoo 


46 Within SEED schoo 


47 Within SEED schoo 


48 Within SEED schoo 


s, the percentage of teachers who were assigned at 
s, the percentage of teachers who were assigned at 
s, the percentage of teachers who were assigned at 


s, the percentage of teachers who were assigned at 


SEED PAK (..e., complete at least 8 resources within one school year). 
°° Within SEED schools, the percentage of teachers who completed SEED PD ranged from 0 to 63% across SEED schools. Six out of 21 schools had a score of 2. 
>! Within SEED schools, the percentage of teachers who completed SEED PD ranged from 6 to 48% across SEED schools. Nine out of 21 schools had a score of 2. 


east one SEED PAK resource during the school year ranged from 0 to 88% 
east one SEED PAK resource during the schoo 


east one SEED PAK resource during the schoo 
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year ranged from 0 to 75% 


year ranged from 0 to 28% 


44 This indicator is measured by the number of teachers who were assigned, self-assigned, or assigned by others, at least one SEED PAK resource during the school year. 
east one SEEK PAK resource during the school year ranged from 0 to 17% 


. None of the SEED schools had a 
. Seven out of 21 schools had a 

. Seven out of 20 schools had a 

. Two out of 20 schools had a 


4 SEED PD included SEED PD offerings (i.e., TLCs, PPS, SEED PAK independent study, book study, and others [i.e., Alt-licensure, induction, EL modules, SEED facilitator]) and 


of at least 8 other and June, IC on percent: ow 
SEED PAK facilitated use | “gradebook” (1) low (0- 13 (14/20 
activities (participation | and 19%): (2) = 
per score must be | facilitated med (20- 10%) 
academic at least 80% use data will 50%); (3) e 
year to be be uploaded high (51- 3 
“complete’) on Google 100%). T4 (15/20 
Drive. = = 
ers 75%) 
Total Score Mean of the two 
for SEED indicators. T1 | 1 
Participation Threshold = 3. 
Component T2 | 1.5 
T3 | 2.5 
T4 | 2 


>? Within SEED schools, the percentage of teachers who completed SEED PD ranged from 10 to 69% across SEED schools. Fourteen out of 20 schools had a score of 2. 
3 Within SEED schools, the percentage of teachers who completed SEED PD ranged from 13 to 100% across SEED schools. Fifteen out of 20 schools had a score of 2. 
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Appendix C. Baseline Equivalence and Impact Analysis 
Outputs and Results 


This appendix shows the statistical outputs from the impact analyses across all six study samples. 
Weights generated from MatchIt were applied in all analyses including the analyses for baseline 
equivalence. Because all outcomes were z-scored, the coefficient was the effect size—the magnitude 
of the difference between SEED and non-SEED schools on the outcomes of interest. Table C1 
shows the statistical outputs of linear regression models for the impact analyses—Goals 5a and 5c. 
Positive findings are highlighted in green. Table C2 provides the sample sizes, adjusted and 
unadjusted group means, and unadjusted standard deviations on outcome measures. Because the 
study samples changed slightly due to missing data in Years 1, 2, and 3, the state of baseline 
equivalence for the Year 1, 2, and 3 study samples are provided in Tables C3, C4, and C5, 
respectively. 


Table Cl. Statistical Outputs of the Impact Analysis 


# | Sample Y1 Outcome Y2 Outcome Y3 Outcome 
B SE SE B SE 

1 | All Students with ELA Outcome 
SmallSch 0.09 | 0.04 | 0.025 0.02 | 0.05 | 0.706 | 0.07 | 0.05 | 0.179 
MID 0.11 | 0.05 | 0.034 0.11 | 0.06 | 0.093 | 0.08 | 0.07 | 0.225 
HIGH 0.02 | 0.05 | 0.664 0.03 | 0.06 | 0.633 | -0.06 | 0.06 | 0.324 
ELMMID 0.14 [0.06 | 0.011 Om oor | Norns) 00s | oor 0515 
MIDHIGH 0.04 | 0.05 | 0.408 0.01 | 0.06 | 0.864 | -0.03 | 0.07 | 0.656 
K12 0.18 | 0.09 | 0.046 SO ee See ae ae 
Remote -0.07 | 0.05 | 0.196 0.09 | 0.06 | 0.183 | 0.04 | 0.07 | 0.596 
MinorityGP 0.04 | 0.05 | 0.511 -0.04 | 0.06 | 0.567 | -0.04 | 0.07 | 0.535 
FRLGP1 0.01 | 0.05 | 0.914 -0.03 | 0.06 | 0.656 | -0.12 | 0.07 | 0.079 
FRLGP2 -0.09 | 0.06 | 0.170 -0.04 | 0.08 | 0.558 | -0.05 | 0.08 | 0.522 
ELGP1 -0.10 | 0.04 | 0.017 -0.09 | 0.05 | 0.079 | -0.02 | 0.05 | 0.658 
ELGP2 0.15 [010 | 0.133 -0.07 | 0.12 | 0.529 | 0.03 | 0.14 | 0.831 
ELA 0.74 | 0.05 | 0.000 0.70 | 0.06 | 0.000 | 0.56 | 0.07 | 0.000 
SEEDSch 0.04 | 0.04 | 0.263 0.06 | 0.05 | 0.197 
R? Change 0.003 0.006 0.02 
Minority Studen 
SmallSch 0.02 | 0.06 | 0.704 |-0.17 | 0.07 [0.032 | -0.23 | 0.08 | 0.007 
BuildingGP 0.13 [0.05 [0.012 |-0.05 |006 [0447 | -0.06 | 0.07 | 0.375 
FRLGP 0.12  |0.07 | 0.089 [000 |009 [0959 [012 |0.10 | 0.207 
MinorityGP 0.02 [0.09 [0853 [0.22 [010 [0042 |[-0.16 [0.11 [0.160 
ELGP 0.13 | 0.06 | 0.024 |-0.05 [0.07 [0454 [005 |0.07 | 0.516 
ELAO 045  |0.12 {0.001 [030 {014 [0042 [056 |0.16 | 0.001 
SEEDSch 020 O06 01002] -0.01 | 0.07 [0933 [004 |o07 | 0.581 
R? Change 0.09 0.00 0.004 


3 | FRL Students with ELA Outcome 


4 The variable Remote was not included in the matching because none of the SEED schools remaining in the final 
sample were in remote areas. Therefore, other non-SEED schools in remote areas were also removed from the dataset 
prior to matching. 
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# | Sample Y1 Outcome Y2 Outcome NANO Cexey ni) 
SE SE B SE 

SmallSch 0.01 0.04 0.798 -0.021 | 0.031 0.499 -0.04 0.03 | 0.244 
MID 0.02 0.04 0.488 0.17 0.03 0.000 0.14 0.03 | 0.000 
HIGH -0.01 0.05 0.893 0.06 0.04 0.166 -0.04 0.04 | 0.323 
ELMMID -0.08 0.06 0.210 -0.15 0.05 0.005 -0.08 0.06 | 0.162 
MIDHIGH 0.03 0.06 0.612 0.17 0.05 0.001 0.09 0.06 | 0.125 
Remote 0.05 0.07 0.547 -0.15 0.06 0.020 0.04 0.07 ‘| 0.568 
MinorityGP -0.07 0.04 0.096 -0.09 0.03 0.010 -0.19 0.04 | 0.000 
FRLGP1 0.06 0.05 0.157 0.08 0.04 0.040 0.07 0.05 | 0.155 
FRLGP2 0.00 0.07 0.960 0.14 0.06 0.018 0.16 0.06 | 0.016 
ELGP‘ -0.10 0.04 0.010 -0.15 0.03 0.000 0.00 0.04 | 0.899 
ELGP2 -0.09 0.07 0.196 -0.09 0.06 0.142 0.10 0.07 | 0.143 
ELAo 0.52 0.08 0.000 0.36 0.07 0.000 0.34 0.07 | 0.000 
SEEDSch 0.07 0.05 0.184 0.03 0.05 | 0.564 
R? Change 0.006 0.01 0.001 
All Students with Math Outcome 
SmallSch 0.07 0.04 0.127 0.02 0.05 0.663 0.04 0.06 | 0.542 
MID 0.02 0.05 0.732 0.12 0.06 0.037 0.10 0.07 | 0.150 
HIGH 0.00 0.05 0.955 0.04 0.05 0.455 0.03 0.07 | 0.617 
ELMMID 0.18 0.06 0.003 0.14 0.06 0.031 0.13 0.08 | 0.104 
MIDHIGH 0.04 0.05 0.432 -0.02 0.05 0.763 0.04 0.06 | 0.496 
K12 0.17 0.11 0.125 0.13 0.12 0.264 0.04 0.14 | 0.775 
Remote -0.11 0.05 0.028 0.09 0.06 0.106 0.03 0.07 | 0.722 
MinGP1 0.00 0.06 0.995 -0.07 0.06 0.231 -0.05 0.07 | 0.470 
MinGP2 0.01 0.11 0.925 -0.03 0.12 0.810 -0.03 0.14 | 0.826 
FRLGP1 -0.10 0.06 0.086 -0.06 0.06 0.374 -0.10 0.08 | 0.206 
FRLGP2 -0.18 0.08 0.022 -0.04 0.08 0.631 -0.07 0.10 | 0.500 
ELGP1 -0.04 0.05 0.439 0.02 0.05 0.649 0.01 0.07 ‘| 0.841 
ELGP2 -0.11 0.09 0.272 -0.11 0.10 0.278 0.09 0.13 | 0.500 
Matho 0.78 0.06 0.000 0.83 0.06 0.000 0.66 0.07 | 0.000 
SEEDSch 0.04 0.04 0.337 0.02 0.04 0.634 0.03 0.05 | 0.634 
R? Change 0.002 0.00 0.001 
Minority Studen 
SmallSch -0.04 0.06 0.437 0.04 0.07 0.587 -0.09 0.09 | 0.304 
BuildingGP 0.04 0.04 0.303 0.02 0.05 0.704 -0.06 0.06 | 0.293 
MinorityGP 0.05 0.04 0.274 -0.09 0.05 0.075 0.24 0.07 ‘| 0.001 
FRLGP -0.13 0.05 0.017 0.01 0.06 0.892 -0.23 0.08 | 0.005 
ELGP -0.03 0.06 0.615 -0.08 0.07 0.242 0.05 0.09 | 0.623 
Matho 0.36 0.12 0.004 0.69 0.14 0.000 0.39 0.18 | 0.041 
SEEDSch 0.03 0.06 0.656 0.11 0.07 | 0.152 
R? Change 0.03 0.001 0.03 
FRL Students with Math Outcome 
SmallSch -0.03 0.03 0.293 0.06 0.04 0.143 0.08 0.03 | 0.018 
MID -0.01 0.03 0.820 0.02 0.04 0.624 -0.02 0.03 | 0.620 
HIGH 0.02 0.04 0.574 0.07 0.04 0.103 -0.06 0.04 | 0.082 
ELMMID -0.11 0.04 0.013 -0.21 0.05 0.000 -0.08 0.05 | 0.089 
MIDHIGH -0.17 0.05 0.001 0.03 0.06 0.575 -0.08 0.05 | 0.124 
Remote 0.05 0.07 0.444 -0.06 0.08 0.460 0.20 0.07 | 0.003 
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# | Sample Y1 Outcome Y2 Outcome Y3 Outcome 
SE SE B SE 
MinGP‘ -0.07 0.04 0.060 -0.01 0.05 0.814 -0.02 0.04 | 0.621 
MinGP2 -0.13 0.08 0.084 -0.20 0.09 0.037 -0.05 0.08 | 0.494 
FRLGP1 0.00 0.04 0.987 0.12 0.05 0.008 0.06 0.04 | 0.146 
FRLGP2 0.06 0.05 0.231 0.33 0.06 0.000 0.06 0.06 | 0.384 
ELGP‘ -0.17 0.03 0.000 -0.09 0.04 0.014 -0.04 0.03 | 0.196 
ELGP2 -0.14 0.05 0.008 -0.19 0.06 0.002 -0.19 0.05 | 0.001 
Matho 0.58 0.06 0.000 0.62 0.07 0.000 0.42 0.06 | 0.000 
SEEDSch 0.06 0.05 0.257 0.03 0.05 | 0.535 
R? Change 0.002 0.006 0.001 


Table C2. Sample Sizes, Adjusted and Unadjusted Group Means, and Unadjusted 
Standard Deviations of Outcome Measures 


SEED Schools Non-SEED Schools 
n_ Adjusted- Unadjusted- SD n_ Adjusted- Unadjusted- SD 
M M M M 

All Students with ELA 
Outcome 
Y1 ELA 19 0.01 0.02 0.42 | 80 -0.03 0.00 0.33 
Y2 ELA 19 0.03 0.00 0.39 | 79 -0.03 -0.03 0.31 
Y3 ELA 18 0.08 0.05 0.33 | 80 -0.02 -0.05 0.30 
Minority Students with 
st OTT Kore) a1 
Y1 ELA : -0.11 -0.14 0.33 = -0.31 -0.39 0.20 
Y2 ELA 0.00 -0.20 0.35 0.01 -0.23 0.22 
Y3 ELA -0.08 -0.23 0.25 -0.12 -0.30 0.24 
Outcome 
Y1 ELA -0.06 -0.30 0.38 | 154 -0.13 -0.34 0.22 
Y2 ELA -0.14 -0.30 0.28 | 151 -0.22 -0.37 0.21 
Y3 ELA -0.22 -0.35 0.23 | 136 -0.25 -0.38 0.21 
Outcome 
Y1 Math 0.09 0.03 0.41 0.05 -0.04 0.37 
Y2 Math -0.02 -0.02 0.42 -0.04 -0.07 0.36 
Y3 Math 7 -0.01 -0.01 0.38 73 -0.04 -0.08 0.31 
Minority Students with 
Math Outcome 
Y1 Math ; -0.11 -0.18 0.24 = -0.20 -0.26 0.17 
Y2 Math -0.02 -0.19 0.38 -0.05 -0.22 0.23 
Y3 Math -0.17 -0.23 0.28 -0.28 -0.34 0.20 
Outcome 
Y1 Math 0.06 -0.34 0.36 | 238 0.00 -0.36 0.26 
Y2 Math -0.13 -0.33 0.27 | 244 -0.22 -0.40 0.26 
Y3 Math : -0.21 -0.35 0.25 | 206 -0.24 -0.40 0.20 


Appendix C-3 


Table C3. Baseline Characteristics of the Yl Study Samples 


Standardized Mean 
Difference 


Variance of the 
PS ratio 


Study SEED Schools Non-SEED Schools 
Sample N M SD N M SD 
1 | All Students with ELA Outcome 
SmallSch 19 | 0.68 | 0.48] 80 | 0.66 | 0.48 
MID 19 | 0.16 | 0.37 | 80 | 0.14 | 0.35 
HIGH 19 | 0.16 | 0.37] 80 | 0.15 | 0.36 
ELMMID 19) NO 032 FS0| Oa ass 
MIDHIGH 19 | 0.16 | 0.37 | 80 | 0.17 | 0.37 
K12 19 | 0.05 | 0.23] 80 | 0.03 | 0.18 
Remote 19 | 0.16 | 0.37| 80 | 0.14 | 0.34 
MinorityGP [19] 0.26 |0.45| 80 | 0.25 | 0.44 
FRLGP1 19 | 0.58 | 0.51] 80 | 0.54 | 0.50 
FRLGP2 19 | 0.21 |0.42| 80 | 0.23 | 0.43 
ELGP1 19| 042 |0.51| 80 | 0.41 | 0.49 
ELGP2 19 | 0.05 | 0.23] 80 | 0.03 | 0.18 
ELAy 19 | -0.02 |0.45| 80 | 0.01 | 0.38 
2 | Minority Students with ELA Outcome 
SmallSch 7 | 0.14 | 0.38] 43 | 0.15 | 0.36 
BuildingGP_ | 7 | 0.71 | 0.49| 43 | 0.60 | 0.50 
FRLGP 7 | 043 | 053] 43 | 0.44 | 0.50 
MinorityGP | 7 | 0.29 |0.49| 43 | 0.35 | 0.48 
ELGP 7 | 0.57 | 053] 43 | 0.68 | 0.47 
ELAg TOT 064s 023 a oe 
3 | FRL Students with ELA Outcome 
SmallSch 12 | 0.50 | 0.52 | 154] 0.70 | 0.46 
MID 12 | 0.25 | 0.45 | 154| 0.27 | 0.45 
HIGH 12 | 0.08 | 0.29 | 154] 0.13 | 0.34 
ELMMID 12 | 0.08 | 0.29 | 154] 0.09 | 0.28 
MIDHIGH 12 | 0.08 | 0.29 | 154] 0.08 | 0.27 
Remote 12 | 0.00 | 0.00 | 154] 0.05 | 0.22 
MinorityGP [12] 0.33 | 0.49 | 154] 0.41 | 0.49 
FRLGP1 12 | 0.58 | 0.51 | 154] 0.65 | 0.48 
FRLGP2 12 | 0.08 | 0.29 | 154] 0.12 | 0.33 
ELGP1 12 | 0.50 | 0.52 | 154 | 0.50 | 0.50 
ELGP2 12 | 0.08 | 0.29 | 154] 0.06 | 0.24 
ELAg 12 | -0.39 | 0.32 | 154] -0.35 | 0.21 
4 | All Students with Math Outcome 
SmallSch 18 | 0.67 |0.49| 71 | 0.76 | 0.43 
MID 185 Oey Ois8) iz Ss 
HIGH 18 | 0.17 |0.38| 71 | 0.16 | 0.37 
ELMMID Aen ON Oe 2 er Oe 
MIDHIGH 18 | 0.17 | 0.38 | 71 | 0.21 | 0.41 
K12 18) O00 oon a oss RO 
Remote 18 | 0.11 | 0.32] 71 | 0.14 | 0.35 
MinGP1 18) O28 046) 7 26 ora 
MinGP2 18 | 0.00 | 0.00] 71 | 0.03 | 0.18 
FRLGP1 18 | 0.61 |0.50| 71 | 063 | 0.49 


0.94 
(0.017 / 0.018) 


1.35 
(0.057 / 0.042) 


0.47 
(0.023 / 0.049) 


0.97 
(0.034 / 0.035) 


0.05 


0.06 


0.02 


-0.06 


-0.02 


0.09 


0.07 


0.03 


0.08 


-0.05 


0.03 


0.09 


-0.08 


-0.01 


0.24 


-0.02 


0.13 


-0.23 


0.11 


-0.41* 


-0.05 


-0.14 


-0.02 


0.01 


-0.32" 


-0.16 


0.13 


-0.12 


0.00 


0.08 


-0.14 


-0.21 


0.07 


0.01 


0.02 


-0.10 


0.25 


-0.09 


0.03 


-0.26 


-0.04 


Appendix C-4 


Variance of the 


Standardized Mean 


Study SEED Schools Non-SEED Schools 
Sample M SD M SD 
FRLGP2 18 | 0.17 | 0.38) 71 0.20 0.40 
ELGP1 18 | 0.39 | 0.50) 71 0.34 0.48 
ELGP2 18 | 0.06 | 0.24) 71 0.04 0.21 
Matho 18 | -0.01 | 0.41 | 71 -0.06 | 0.38 
SmallSch 7 | 0.14 | 0.38 | 50 0.08 0.28 
BuildingGP 7 | 0.71 | 0.49 | 50 0.78 0.42 
MinorityGP 7 | 0.43 | 0.53 | 50 0.51 0.50 
FRLGP 7 | 0.29 | 0.49 | 50 0.32 0.47 
ELGP 7 | 0.57 | 0.53 | 50 0.44 0.50 
Matho 7 | -0.15 | 0.37 | 50 | -0.16 0.22 
6 | FRL Students with Math Outcome 
SmallSch 12 | 0.50 | 0.52 | 238 | 0.53 0.50 
MID 12 | 0.25 | 0.45 | 238} 0.25 0.44 
HIGH 12 | 0.08 | 0.29 | 238} 0.16 0.37 
ELMMID 12 | 0.08 | 0.29 | 238} 0.14 0.35 
MIDHIGH 12 | 0.08 | 0.29 | 238} 0.07 0.25 
Remote 12 | 0.00 | 0.00 | 238 | 0.04 0.19 
MinGP1 12 | 0.33 | 0.49 | 238} 0.26 0.44 
MinGP2 12 | 0.00 | 0.00 | 238} 0.03 0.17 
FRLGP'1 12 | 0.58 | 0.51 | 238} 0.58 0.49 
FRLGP2 12 | 0.08 | 0.29 | 238} 0.13 0.33 
ELGP'1 12 | 0.50 | 0.52 | 238} 0.43 0.50 
ELGP2 12 | 0.08 | 0.29 | 238} 0.07 0.26 
Matho 12 | -0.42 | 0.33 | 238) -0.39 | 0.28 


PS ratio 


0.96 
(0.046 / 0.048) 


0.97 
(0.035 / 0.036) 


Difference 
-0.08 


0.11 


0.05 


0.11 


0.18 


0.13 


-0.16 


-0.07 


0.25 


0.04 


-0.06 


-0.01 


-0.24 


-0.18 


0.06 


-0.27" 


0.15 


-0.25 


0.01 


-0.14 


0.13 


0.03 


-0.08 


* The standardized mean difference on the variable was greater than 0.25. 


Table C4. Baseline Characteristics of the Y2 Study Samples 


Variance of the 
PS ratio 


Standardized Mean 
Difference 


/1/ AllStudentswithELAOutcome 


Study SEED Schools Non-SEED Schools 
Sample N M SD N M SD 
SmallSch 19 | 068 | 0.48 | 79 | 0.66 | 0.48 
MID 19) 0.16) 0:37 | 79 | 0.14. | 0.35 
HIGH 19 | 0.16 | 0.37 | 79 | 0.15 | 0.36 
ELMMID 19) Ose O25 Toes a 32 
MIDHIGH 19 | 0.16 | 0.37 | 79 | 0.17 | 0.38 
K12 19 | 0.05 | 0.23 | 79 | 0.04 | 0.19 
Remote 19 | 0.16 | 0.37 | 79 | 0.13 | 0.33 
MinorityGP | 19} 0.26 | 0.45| 79 | 0.25 | 0.44 
FRLGP1 19)|20:58 0:51) 5790.53 2050 
FRLGP2 19 | 0.21 | 0.42) 79 | 0.24 | 0.43 
ELGP1 19 | 0.42 | 0.51 | 79 | 0.41 | 0.50 
ELGP2 19 | 0.05 | 0.23 | 79 | 0.04 | 0.19 
ELAo 19 | -0.02 | 0.45 | 79 | 0.02 | 0.38 


0.94 
(0.017 / 0.018) 


0.06 


0.06 


0.02 


-0.03 


-0.02 


0.09 


0.09 


0.02 


0.09 


-0.06 


0.02 


0.09 


-0.10 
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Study SEED Schools Non-SEED Schools _ Variance of the Standardized Mean 
Sample N M SD N M AyD) PS ratio Difference 
2 | Minority Students with ELA Outcome * 
SmallSch 7 | 0.14 | 0.38 | 42 | 0.15 | 0.36 1.34 -0.02 
BuildingGP 7 | 0.71 | 0.49 | 42 | 0.61 | 0.49 (0.057 / 0.042) 0.21 
FRLGP 7 | 0.43 | 0.53 | 42 | 0.42 | 0.50 0.01 
MinorityGP 7 | 0.29 | 0.49 | 42 | 0.36 | 0.49 -0.15 
ELGP 7 | 0.57 | 0.53 | 42 | 0.70 | 0.46 -0.25 
ELAo 7 | 0:21 | 0.36) 42. | -0.25. | 0.23 0.13 
3 | FRL Students with ELA Outcome 
SmallSch 15 | 0.60 | 0.51 | 151} 0.69 | 0.46 0.51 -0.19 
MID 15 | 0.20 | 0.41 | 151] 0.26 | 0.44 (0.022 / 0.043) -0.13 
HIGH 15 | 0.13 | 0.35 | 151] 0.12 | 0.33 0.03 
ELMMID 15) (203 02085: 151) | 009 70:29 0.13 
MIDHIGH 15 | 0.13 | 0.35 | 151 | 0.09 | 0.29 0.13 
Remote 1500.07 0:26 151) | 0.05 40.22 0.07 
MinorityGP | 15 | 0.33 | 0.49 | 151 | 0.43 | 0.50 -0.19 
FRLGP1 15 | 0.60 | 0.51 | 151 | 0.64 | 0.48 -0.09 
FRLGP2 15 | 0.13 | 0.35 | 151 | 0.12 | 0.33 0.03 
ELGP1 15: 0.47 90:52 | 151) | 051) | 0:50 -0.08 
ELGP2 15 | 0.07 | 0.26 | 151 | 0.07 | 0.26 -0.03 
ELAo 15 | -0.40 | 0.29 | 151 | -0.36 | 0.21 -0.15 
4 All Students with Math Outcome 
SmallSch 18 | 0.67 | 0.49} 70 | 0.76 | 0.43 0.97 -0.21 
MID 18) O47 | 0.385) 705) 0.140 | 0535 (0.034 / 0.035) 0.06 
HIGH 18 | 0.17 | 0.38 | 70 | 0.16 | 0.37 0.01 
ELMMID 18) 0110.32) |) 200 | 009 50:28 0.05 
MIDHIGH 18 | 0.17 | 0.38 | 70 | 0.21 | 0.41 -0.11 
K12 18 | 0.00 | 0.00; 70 | 0.03 | 0.17 -0.25 
Remote 18 | 0.11 | 0.32) 70 | 0.13 | 0.34 -0.06 
MinGP‘ 18 | 0.28 | 0.46 | 70 | 0.27 | 0.44 0.03 
MinGP2 18 | 0.00 | 0.00] 70 | 0.03 | 0.18 -0.26* 
FRLGP1 18 | 0.61 | 0.50} 70 | 0.63 | 0.49 -0.04 
FRLGP2 18 | 0.17 | 0.38 | 70 | 0.20 | 0.40 -0.09 
ELGP1 18 | 0.39 | 0.50| 70 | 0.34 | 0.48 0.10 
ELGP2 18 | 0.06 | 0.24} 70 | 0.05 | 0.21 0.05 
Matho 18 | -0.01 | 0.41 | 70 | -0.05 | 0.38 0.10 
5 | Minority Students with Math Outcome 
SmallSch 7 | 0.14 (0.38; 50 | 0.08 | 0.28 0.95 0.18 
BuildingGP 0.71 | 0.49 | 50 | 0.78 | 0.42 (0.046 / 0.048) -0.13 
MinorityGP 7 | 0.43 | 0.53 | 50 | 0.51 | 0.51 -0.15 
FRLGP 7 | 0.29 | 0.49 | 50 | 0.32 | 0.47 -0.07 
ELGP 7 | 0.57 | 0.53 | 50 | 0.44 | 0.50 0.25 
Matho POS 00.87 500 OZ | G22 0.04 


The variable Remote was not included in the matching because none of the SEED schools remaining in the final 
sample were in remote areas. Therefore, other non-SEED schools in remote areas were also removed from the dataset 
prior to matching. 
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Variance of the 


PS ratio 


Standardized Mean 
Difference 


Study SEED Schools Non-SEED Schools 
Sample N M SD N M SD 
6 | FRL Students with Math Outcome 
SmallSch 15 | 0.60 | 0.51 | 244] 0.54 0.50 
MID 15 | 0.20 | 0.41 | 244} 0.25 0.43 
HIGH 15 | 0.13 | 0.35 | 244} 0.16 0.37 
ELMMID 15 | 0.13 | 0.35 | 244} 0.14 0.35 
MIDHIGH 15 | 0.13 | 0.35 | 244 | 0.09 0.29 
Remote 15 | 0.07 | 0.26 | 244} 0.03 0.18 
MinGP‘ 15 | 0.33 | 0.49 | 244 | 0.28 0.45 
MinGP2 15 | 0.00 | 0.00 | 244} 0.03 0.17 
FRLGP'1 15 | 0.60 | 0.51 | 244} 0.59 0.49 
FRLGP2 15 | 0.13 | 0.35 | 244} 0.12 0.33 
ELGP1 15 | 0.47 | 0.52 | 244} 0.42 0.50 
ELGP2 15 | 0.07 | 0.26 | 244} 0.10 0.30 
Matho 15 | -0.44 | 0.30 | 244) -0.39 0.27 


1.00 
(0.034 / 0.034) 


0.12 


-0.11 


-0.07 


-0.01 


0.14 


0.14 


0.11 


-0.24 


0.02 


0.03 


0.09 


-0.11 


-0.18 


* The standardized mean difference on the variable was greater than 0.25. 


Table C5. Baseline Characteristics of the Y3 Study Samples 


Study SEED Schools Non-SEED Schools __ Variance of the Standardized Mean 
Sample N M SD N M AY?) PS ratio Difference 
1 All Students with ELA Outcome 
SmallSch 18 | 0.67 | 0.49 | 80 | 0.66 | 0.48 0.89 0.01 
MID 16 BO. SE0:38) | O0u | 0 as 0:35 (0.017 / 0.019) 0.09 
HIGH 18 | 0.17 | 0.38 | 80 | 0.15 | 0.36 0.05 
ELMMID Go 2021 032 ee | Onl 0.32 -0.01 
MIDHIGH 18 | 0.17 | 0.38 | 80 | 0.16 | 0.37 0.00 
K12 18 | 0.06 | 0.24 | 80 | 0.03 | 0.18 0.11 
Remote 18 | 0.17 | 0.38 | 80 | 0.14 | 0.35 0.08 
MinorityGP 18 | 0.22 | 0.43 | 80 | 0.26 | 0.44 -0.09 
FRLGP1 18 | 0.61 | 0.50 | 80 | 0.54 | 0.50 0.14 
FRLGP2 18 | 0.17 | 0.38 | 80 | 0.23 | 0.43 -0.16 
ELGP1 18 | 0.44 | 0.51 | 80 | 042 | 0.50 0.05 
ELGP2 18 | 0.00 | 0.00 | 80 | 0.03 | 0.18 -0.21 
ELAo 18 | 0.01 | 0.44 | 80 | 0.01 0.38 0.00 
SmallSch 7 | 0.14 | 0.38 | 42 | 0.15 | 0.36 1.34 -0.02 
BuildingGP 70 020-495 425) 064 0.49 (0.057 / 0.042) 0.21 
FRLGP 7 | 0.43 | 0.53 | 42 | 0.42 | 0.50 0.01 
MinorityGP 7 | 0.29 | 0.49 | 42 | 0.36 | 0.49 -0.15 
ELGP 7 | 0.57 | 0.53 | 42 | 0.70 | 0.46 -0.25 
ELAo 7. || -0.21 | 0.36 | 42 | -0:25 | 0.23 0.13 
3 FRL Students with ELA Outcome 
SmallSch 12 | 0.50 | 0.52 | 136 | 0.66 | 0.48 0.53 -0.32 


6 The variable Remote was not included in the matching because none of the SEED schools remaining in the final 
sample were in remote areas. Therefore, other non-SEED schools in remote areas were also removed from the dataset 


prior to matching. 
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# Study SEED Schools Non-SEED Schools _ Variance of the Standardized Mean 
SET) (2) M AY?) M AD) EL) Difference 
MID 12 | 0.25. 0:45 1, 136)| 0.28 | 0.45 (0.024 / 0.045) -0.06 
HIGH 12 | 0.17 | 0.39 | 136 | 0.13 | 0.33 0.11 
ELMMID 12) 20.080)|0:29 136) (0105030 -0.05 
MIDHIGH 12 | 0.08 | 0.29 | 136 | 0.09 | 0.29 -0.03 
Remote 12) OOe POC T1365 | a0 0es 023 -0.34 
MinorityGP 12 | 0.25 | 0.45 | 136 | 0.47 | 0.50 -0.45 
FRLGP1 12 | 0.67 | 0.49 | 136 | 0.70 | 0.46 -0.06 
FRLGP2 12 | 0.00 | 0.00 | 136 | 0.14 | 0.34 -0.56* 
ELGP1 121/050) 052) 138 | 0:53 (80.50 -0.07 
ELGP2 12 | 0.00 | 0.00 | 136 | 0.07 | 0.26 -0.39 
ELAo 12 | -0.38 | 0.31 | 136 | -0.38 | 0.21 -0.02 
SmallSch 17 | 0.65 | 049 | 69 | 0.76 | 0.43 0.92 -0.24 
MID 17) O18. 0:38 69" ||, 04384 0.84 (0.034 / 0.037) 0.13 
HIGH 17 | 0.18 | 0.39 | 69 | 0.17 | 0.37 0.03 
ELMMID 1 O22 0335/2699 | 010 20.58 0.07 
MIDHIGH 17 | 0.18 | 0.39 | 69 | 0.21 0.41 -0.09 
K12 17 | 0.00 | 0.00 | 69 | 0.03 | 0.17 -0.25 
Remote 17 | 0.12 | 0.33 | 69 | 0.12 | 0.32 0.01 
MinGP1 17 | 0.24 | 0.44 | 69 | 0.27 | 0.45 -0.08 
MinGP2 17 | 0.00 | 0.00 | 69 | 0.03 | 0.18 -0.27* 
FRLGP1 17 | 0.65 | 049 | 69 | 0.62 | 0.49 0.05 
FRLGP2 17 | 0.12 | 0.33 | 69 | 0.20 | 0.41 -0.23 
ELGP1 17 | 041 9) 0-51) | 69. || 0.35 | 0.48 0.13 
ELGP2 17 | 0.00 | 0.00 | 69 | 0.05 | 0.21 -0.31 
Matho 17 | 0.02 | 0.39 | 69 | -0.04 | 0.38 0.16 
SmallSch 7 | 0.14 | 0.38 | 50 | 0.08 | 0.28 0.96 0.18 
BuildingGP 7 | 0:71 | 0.49 50 | 078 | 0:42 (0.046 / 0.048) -0.13 
MinorityGP 7 | 0.43 | 0.53 | 50 | 0.51 0.51 -0.15 
FRLGP 70.29) | 0.49 || 50. | 0.382 0:47 -0.07 
ELGP 7 | 0.57 | 0.53 | 50 | 0.44 | 0.50 0.25 
Matho TAOS 20372505 | 0st 0:22 0.04 

6 FRL Students with Math Outcome 
SmallSch 12 | 0.50 | 0.52 | 206 | 0.54 | 0.50 1.00 -0.08 
MID 12 | 0.25 | 0.45 | 206 | 0.28 | 0.45 (0.035 / 0.035) -0.07 
HIGH 12 | 0.17 | 0.39 | 206 | 0.17 | 0.38 -0.01 
ELMMID 12 ON ee O29 2065 2 ess -0.12 
MIDHIGH 12 | 0.08 | 0.29 | 206 | 0.08 | 0.27 0.02 
Remote 12 | 0.00 | 0.00 | 206 | 0.04 | 0.20 -0.21 
MinGP‘ 12 | 0.25 | 0.45 | 206 | 0.30 | 0.46 -0.12 
MinGP2 12 | 0.00 | 0.00 | 206 | 0.03 | 0.18 -0.19 
FRLGP1 12 | 0.67 | 0.49 | 206 | 0.66 | 0.47 0.01 
FRLGP2 12) O00 000206 | 040 031 -0.35* 
ELGP1 12 | 0.50 | 0.52 | 206 | 0.47 | 0.50 0.06 
ELGP2 12 | 0.00 | 0.00 | 206 | 0.09 | 0.28 -0.31* 
Matho 12 | -0.41 | 0.32 | 206 | -0.43 | 0.27 0.08 
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# Study SEED Schools Non-SEED Schools __ Variance of the Standardized Mean 


Sample N M SD N M SD PS ratio Difference 
* The standardized mean difference on the variable was greater than 0.25. 
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